Knowledge base construction method and system based on large language model

Abstract

The invention provides a knowledge base construction method and system based on a large language model, and the method comprises the steps: carrying out the preprocessing of an original text, and obtaining a plurality of original text paragraphs; performing deep elaboration on each original text paragraph through a language enhancement model to generate detailed text content, converting the detailed text content into a high-dimensional vector through a text vector model, and matching the high-dimensional vector with one original text paragraph; the high-dimensional vectors and the corresponding original text paragraphs are stored in a database, and the database supports vector indexing and retrieval operation. Through the method and the device, the text processing yield can be improved, certain manpower and material resources can be saved, and meanwhile, the database maintenance difficulty can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge base construction, and in particular to a knowledge base construction method and system based on a large language model. Background Art

[0002] At present, the construction of knowledge base mainly relies on manual methods to segment and summarize text documents, store text content in the database, and directly present the searched content on the front end through keyword retrieval, or add fixed content introductions before and after to enhance text comprehension.

[0003] The construction objects of traditional knowledge bases mainly deal with text data, and their ability to integrate and process multimodal data such as images and audio is limited; the construction process of traditional knowledge bases requires manual extraction, induction and storage of knowledge, which is time-consuming, labor-intensive and error-prone; at the same time, with the continuous updating and changes of knowledge, the maintenance of the knowledge base has also become a huge challenge. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a knowledge base construction method and system based on a large language model to solve the deficiencies in the prior art.

[0005] To achieve the above object, the present invention provides a method for constructing a knowledge base based on a large language model, the method comprising:

[0006] Preprocessing the original text to obtain multiple original text paragraphs;

[0007] Performing in-depth elaboration on each of the original text paragraphs through a language enhancement model to generate detailed text content, and converting the detailed text content into a high-dimensional vector through a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0008] Storing the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations;

[0009] Converting the query question into a question vector, comparing the question vector with the text vector in the database, and selecting N relevant text paragraphs related to the question vector from the database according to the comparison result;

[0010] The step of selecting N relevant text paragraphs related to the question vector from the database according to the comparison result comprises:

[0011] Calculating the question vector and each high-dimensional vector in the database one by one by a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector;

[0012] N relevant text paragraphs related to the question vector are screened out based on the multiple similarity scores.

[0013] Preferably, after screening out N relevant text paragraphs related to the question vector based on the plurality of similarity scores, the method further comprises:

[0014] The N relevant text paragraphs are sorted by a sorting model, a weight value is assigned to each relevant text paragraph according to the sorting result, and the final relevant text paragraph is screened out based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

[0015] Preferably, after the final relevant text paragraphs are screened out based on the weight values, the method further comprises:

[0016] Inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0017] The target answer is displayed through a user interface.

[0018] Preferably, the step of preprocessing the original text to obtain a plurality of original text paragraphs includes:

[0019] The original text is obtained, and the original text is segmented using natural language processing technology to obtain multiple independent original text paragraphs.

[0020] To achieve the above object, the present invention also provides a knowledge base construction system based on a large language model, which is used to implement the knowledge base construction method based on a large language model described above, and the system includes:

[0021] A preprocessing module, used for preprocessing the original text to obtain multiple original text paragraphs;

[0022] A conversion module, configured to perform an in-depth explanation of each of the original text paragraphs through a language enhancement model to generate detailed text content, and convert the detailed text content into a high-dimensional vector through a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0023] A storage module, used to store the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations;

[0024] A comparison module, used to convert the query question into a question vector, compare the question vector with the text vector in the database, and select N relevant text paragraphs related to the question vector from the database according to the comparison result;

[0025] The comparison module comprises:

[0026] A calculation unit, used to calculate the question vector and each high-dimensional vector in the database one by one by using a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector;

[0027] A screening unit is used to screen out N relevant text paragraphs related to the question vector based on the multiple similarity scores.

[0028] Preferably, after the screening unit, the system further comprises:

[0029] A sorting unit is used to sort the N related text paragraphs through a sorting model, assign a weight value to each related text paragraph according to the sorting result, and screen out the final related text paragraphs based on each weight value, wherein the weight value is used to reflect the degree of relevance between the related text paragraph and the corresponding question to be queried.

[0030] Preferably, after the sorting unit, the system further includes:

[0031] A generating unit, used for inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0032] A display unit is used to display the target answer through a user interface.

[0033] Preferably, the preprocessing module comprises:

[0034] The segmentation unit is used to obtain the original text and segment the original text by natural language processing technology to obtain multiple independent original text paragraphs.

[0035] The beneficial effects of the present invention are as follows: by preprocessing the original text, a plurality of original text paragraphs are obtained, each original text paragraph is deeply elaborated through a language enhancement model to generate detailed text content, and the detailed text content is converted into a high-dimensional vector through a text vector model, and then the high-dimensional vector and the corresponding original text paragraph are stored in a database. Different from the prior art, the productivity of text processing can be improved, which is conducive to saving certain manpower and material resources, and is also conducive to reducing the difficulty of database maintenance. The application of the language enhancement model can further enhance the context of the text, making the information in the database more accurate and complete, which is conducive to improving the overall quality of the database and also provides a more solid foundation for subsequent knowledge retrieval.

[0036] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a method for constructing a knowledge base based on a large language model provided in the first embodiment of the present invention;

[0038] Figure 2 This is a structural block diagram of a knowledge base construction system based on a large language model provided in the second embodiment of the present invention.

[0039] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0041] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0042] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0043] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0044] See also Figure 1 , is a flowchart of a method for constructing a knowledge base based on a large language model in a first embodiment of the present invention, and the method for constructing a knowledge base based on a large language model comprises the following steps:

[0045] Step S101, preprocessing the original text to obtain a plurality of original text paragraphs;

[0046] The step of preprocessing the original text to obtain a plurality of original text paragraphs includes:

[0047] The original text is obtained, and the original text is segmented using natural language processing technology to obtain multiple independent original text paragraphs.

[0048] It should be noted that the input original text (such as books, articles, reports and other documents in various formats) is cut into a series of independent text paragraphs through natural language processing technology. This step helps to break down large blocks of text content into smaller units that are easier to process.

[0049] Step S102, deeply elaborating each of the original text paragraphs through a language enhancement model to generate detailed text content, and converting the detailed text content into a high-dimensional vector through a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0050] Among them, the language enhancement model can understand and generate text content that is similar to the original text, contains more details or is easier to understand. In this step, the original text paragraph is deeply explained through the language enhancement model (such as the GPT series), which can retain and enhance the context of the original text paragraph. This not only helps to improve the accuracy of the database, but also enhances the readability and usability of the data in the database.

[0051] It should be noted that advanced text vector models (such as embeddings) are used to convert text paragraphs processed by the language enhancement model into high-dimensional vectors. These high-dimensional vectors can capture the semantic features and contextual relationships of the detailed text content, providing a basis for subsequent knowledge retrieval.

[0052] Step S103: storing the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations.

[0053] The database should be selected to support efficient vector indexing and retrieval operations so as to quickly find knowledge related to the query.

[0054] Through the above steps, the original text is preprocessed to obtain multiple original text paragraphs, and each original text paragraph is deeply elaborated through the language enhancement model to generate detailed text content, and the detailed text content is converted into a high-dimensional vector through the text vector model, and then the high-dimensional vector and the corresponding original text paragraph are stored in the database. Different from the existing technology, it can improve the productivity of text processing, which is conducive to saving certain manpower and material resources, and is also conducive to reducing the difficulty of database maintenance. The application of the language enhancement model can further enhance the context of the text, making the information in the database more accurate and complete, which is conducive to improving the overall quality of the database and providing a more solid foundation for subsequent knowledge retrieval.

[0055] In some embodiments, the method further comprises:

[0056] Pre-train based on the large language model to obtain the initial generation model;

[0057] Among them, first, the traditional large language model is used as the technical foundation for basic pre-training, so that the initial generation model can generate reasonable text. The steps of pre-training based on the large language model are specifically to train the traditional large language model through large-scale historical text data to learn the basic structure and vocabulary of the language.

[0058] Customizing a reward function, and performing reinforcement training on the initial generation model through the reward function to obtain a preliminary language reinforcement model;

[0059] In order to make the text generated by the initial generative model more in line with the target, a reward function is designed. The reward function takes into account multiple dimensions, such as the accuracy, fluency, information richness and relevance of the generated answers to the expected goals. For example, in question-answering tasks, the model can obtain corresponding rewards based on factors such as the accuracy, completeness and interpretability of the answers.

[0060] Specifically, during the training process, the model receives rewards or penalties based on the generated text, and then adjusts the generation strategy. This process is achieved through methods such as "strategy gradient" or "Q-learning" to maximize the final long-term reward.

[0061] The prepared language reinforcement model is optimized through a feedback mechanism to obtain the required language reinforcement model.

[0062] Among them, after each generated text output, the reinforcement learning mechanism will optimize the model's behavior through feedback (evaluation from the reward function). The model continuously adjusts the strategy in the generation process through trial and error, receiving rewards and penalties to improve the quality and conformity of text generation. Ultimately, after multiple reinforcement learning training and optimization, the model will generate higher quality answers, which not only ensures the fluency of language generation, but also improves the accuracy, richness and reasoning ability of the answers.

[0063] In some embodiments, after storing the high-dimensional vector and the corresponding original text paragraph in a database, the method further includes:

[0064] The question to be queried is converted into a question vector, the question vector is compared with the text vector in the database, and N relevant text paragraphs related to the question vector are screened out from the database according to the comparison result.

[0065] Among them, when the user inputs a question to be queried, the text vector model in step S102 is first used to convert the question to be queried into a question vector. The question vector and the high-dimensional vector are both text vectors. This step ensures that the question to be queried can be compared with the high-dimensional vector in the database in the same semantic space so that the similarity score between them can be calculated. In addition, accurate vectorization processing of the query question and efficient comparison with the text vector in the database are also achieved. Compared with the traditional knowledge retrieval method of searching by keyword matching, it is not only beneficial to improve the accuracy of knowledge retrieval, but also can greatly shorten the retrieval time and improve the retrieval efficiency, thereby improving the user experience.

[0066] It should be noted that, based on the comparison results, N original text paragraphs related to the question to be queried are screened out, and the value of N can be adjusted according to actual needs to balance the retrieval accuracy and recall rate, and the original text paragraphs related to the question to be queried are the relevant text paragraphs.

[0067] In some embodiments, the step of selecting N relevant text paragraphs related to the question vector from the database according to the comparison result includes:

[0068] Calculating the question vector and each high-dimensional vector in the database one by one by a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector;

[0069] N relevant text paragraphs related to the question vector are screened out based on the multiple similarity scores.

[0070] In some embodiments, after selecting N relevant text paragraphs related to the question vector based on the plurality of similarity scores, the method further includes:

[0071] The N relevant text paragraphs are sorted by a sorting model, a weight value is assigned to each relevant text paragraph according to the sorting result, and the final relevant text paragraph is screened out based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

[0072] Among them, the sorting model is a sorting algorithm based on machine learning, and the sorting model can be trained and optimized according to the semantic features of the text, contextual relationships and user feedback information to update the sorting model in real time to improve the accuracy of the sorting model.

[0073] It should be noted that the weight value assigned to the relevant text paragraph can reflect the relevance between the relevant text paragraph and the question to be queried.

[0074] In some embodiments, after the final relevant text paragraphs are screened out based on the weight values, the method further includes:

[0075] Inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0076] Among them, the language enhancement model can comprehensively consider the information of multiple related text paragraphs. Therefore, the final related text paragraphs and corresponding weight values ​​are input into the language enhancement model for language reorganization, so as to obtain a coherent and accurate target answer.

[0077] The target answer is displayed through a user interface.

[0078] The second embodiment of the present application also provides a knowledge base construction system based on a large language model, which is used to implement the above-mentioned first embodiment and preferred implementation mode, which have been described and will not be repeated. As used below, the terms "module", "unit", "subunit" and the like can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0079] like Figure 2 As shown, it is a structural block diagram of the knowledge base construction system based on the large language model in the second embodiment of the present application, such as Figure 2 As shown, the system comprises:

[0080] A preprocessing module 10 is used to preprocess the original text to obtain a plurality of original text paragraphs;

[0081] A conversion module 20, configured to perform an in-depth explanation of each of the original text paragraphs through a language enhancement model to generate detailed text content, and convert the detailed text content into a high-dimensional vector through a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0082] The storage module 30 is used to store the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations.

[0083] In the specific implementation, the original text is preprocessed to obtain multiple original text paragraphs, and each original text paragraph is deeply elaborated through the language enhancement model to generate detailed text content, and the detailed text content is converted into a high-dimensional vector through the text vector model. Then the high-dimensional vector and the corresponding original text paragraph are stored in the database. Different from the existing technology, it can improve the productivity of text processing, help save certain manpower and material resources, and also help reduce the difficulty of database maintenance. The application of the language enhancement model can further enhance the context of the text, making the information in the database more accurate and complete, which is conducive to improving the overall quality of the database and providing a more solid foundation for subsequent knowledge retrieval.

[0084] In some embodiments, after the storage module 30, the system further includes:

[0085] The comparison module is used to convert the query question into a question vector, compare the question vector with the text vector in the database, and select N relevant text paragraphs related to the question vector from the database according to the comparison result.

[0086] In some embodiments, the comparison module includes:

[0087] A calculation unit, used to calculate the question vector and each high-dimensional vector in the database one by one by using a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector;

[0088] A screening unit is used to screen out N relevant text paragraphs related to the question vector based on the multiple similarity scores.

[0089] In some embodiments, after the screening unit, the system further comprises:

[0090] A sorting unit is used to sort the N related text paragraphs through a sorting model, assign a weight value to each related text paragraph according to the sorting result, and screen out the final related text paragraphs based on each weight value, wherein the weight value is used to reflect the degree of relevance between the related text paragraph and the corresponding question to be queried.

[0091] In some embodiments, after the screening unit, the system further comprises:

[0092] A generating unit, used for inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0093] A display unit is used to display the target answer through a user interface.

[0094] In some embodiments, the pre-processing module 10 includes:

[0095] The segmentation unit is used to obtain the original text and segment the original text by natural language processing technology to obtain multiple independent original text paragraphs.

[0096] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a knowledge base based on a large language model, characterized in that: The method comprises: Preprocessing the original text to obtain multiple original text paragraphs; Performing in-depth elaboration on each of the original text paragraphs through a language enhancement model to generate detailed text content, and converting the detailed text content into a high-dimensional vector through a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs; Storing the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations; Converting the query question into a question vector, comparing the question vector with the text vector in the database, and selecting N relevant text paragraphs related to the question vector from the database according to the comparison result; The step of selecting N relevant text paragraphs related to the question vector from the database according to the comparison result comprises: Calculating the question vector and each high-dimensional vector in the database one by one by a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector; N relevant text paragraphs related to the question vector are screened out based on the multiple similarity scores.

2. The method for constructing a knowledge base based on a large language model according to claim 1, characterized in that: After screening out N relevant text paragraphs related to the question vector based on the plurality of similarity scores, the method further includes: The N relevant text paragraphs are sorted by a sorting model, a weight value is assigned to each relevant text paragraph according to the sorting result, and the final relevant text paragraph is screened out based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

3. The method for constructing a knowledge base based on a large language model according to claim 2, characterized in that: After the final relevant text paragraphs are screened out based on the weight values, the method further includes: Inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer; The target answer is displayed through a user interface.

4. The method for constructing a knowledge base based on a large language model according to claim 1, characterized in that: The step of preprocessing the original text to obtain multiple original text paragraphs includes: The original text is obtained, and the original text is segmented using natural language processing technology to obtain multiple independent original text paragraphs.

5. A knowledge base construction system based on a large language model, used to implement the knowledge base construction method based on a large language model according to any one of claims 1 to 4, characterized in that: The system comprises: A preprocessing module, used for preprocessing the original text to obtain multiple original text paragraphs; A conversion module, configured to perform an in-depth explanation of each of the original text paragraphs through a language enhancement model to generate detailed text content, and convert the detailed text content into a high-dimensional vector through a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs; A storage module, used to store the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations; A comparison module, used to convert the query question into a question vector, compare the question vector with the text vector in the database, and select N relevant text paragraphs related to the question vector from the database according to the comparison result; The comparison module comprises: A calculation unit, used to calculate the question vector and each high-dimensional vector in the database one by one by using a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector; A screening unit is used to screen out N relevant text paragraphs related to the question vector based on the multiple similarity scores.

6. The knowledge base construction system based on a large language model according to claim 5 is characterized in that: After the screening unit, the system further comprises: A sorting unit is used to sort the N related text paragraphs through a sorting model, assign a weight value to each related text paragraph according to the sorting result, and screen out the final related text paragraphs based on each weight value, wherein the weight value is used to reflect the degree of relevance between the related text paragraph and the corresponding question to be queried.

7. The knowledge base construction system based on a large language model according to claim 6 is characterized in that: After the sorting unit, the system further includes: A generating unit, used for inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer; A display unit is used to display the target answer through a user interface.

8. The knowledge base construction system based on a large language model according to claim 1, characterized in that: Before the comparison module, the preprocessing module includes: The segmentation unit is used to obtain the original text and segment the original text by natural language processing technology to obtain multiple independent original text paragraphs.

Images