Vector knowledge base construction method and system based on large model

Abstract

According to the method, the current knowledge base vectorization method is limited to classification of words and has limited ability to understand sentences, and information loss caused by single vectorization of large-section text data is considered; the vector knowledge base construction method and system based on the large model, which are more suitable for word embedding model vectorization, are created by utilizing the capacity of the large model. The method comprises the following steps of text segmentation, description information extraction, key description concept extraction, concept data dictionary output and vector knowledge base integration. Wherein the concept data dictionary output comprises concept filtering and merging and concept relationship rule arrangement. The system comprises a text segmentation module, an information description module, a concept extraction module, a concept filtering and merging module, a concept relationship arrangement module, a large model module, a word embedding module and a vector knowledge base integration module.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing, and in particular to a method and system for constructing a vector knowledge base based on a large model. Background Art

[0002] Local knowledge bases are generally used in applications that require relatively professional or confidential data. In them, keyword search, matching, knowledge graphs and other methods are used to retrieve questions to obtain corresponding information. Local knowledge base vectorization is to convert text data in the knowledge base into vector representations to facilitate the processing of machine learning models and natural language processing tasks. Knowledge base vectorization is currently an important technology and application direction for application in combination with large models. The background technology of vectorization mainly involves word embedding and semantic representation technology. Traditional word embedding methods map vocabulary to a low-dimensional vector space by counting the co-occurrence information of words, thereby capturing the semantic relationship between words. However, these methods have limited ability to capture contextual information and are difficult to handle polysemous words and complex contextual dependencies.

[0003] Knowledge base vectorization has broad prospects in text retrieval to obtain relevant text data information, but there are still some limitations in current applications. For example:

[0004] 1. Limitations of the model: Current word embedding models focus more on word-level training. Although they can better distinguish the high-dimensional spatial distribution of words, they are still limited by some semantic merging algorithms and natural non-sentence understanding training models for understanding long sentences.

[0005] 2. Limitations of polysemy and complex text: The text vectorization in the knowledge base is usually not a few simple sentences, but may contain vectorization of a paragraph of text information. For multiple meanings, multiple concepts or multiple pieces of content that may be expressed in a paragraph of text, a single vectorization cannot express them well.

[0006] 3. Lack of concept extraction and logical association: Knowledge base vectorization is more about extracting text information, but does not perform more precise concept extraction. In addition, the extended links in logical causal relationships are also unclear, and retrieval is simply based on the similarity of vectors. These have affected the retrieval effect of the knowledge base to a certain extent.

[0007] Traditional knowledge base vectorization uses text segmentation, usually by the number of words in the text, and then vectorizes the segmented text and stores it in the vector library. When applied, the search question is directly vectorized and the similarity is compared in the vector library. Although it can realize basic search functions, it seems to be unable to cope with the above challenges. Summary of the invention

[0008] The purpose of the present invention is to propose a vector knowledge base construction method and system based on a large model, taking into account the knowledge base vectorization process and the ability limitations of the word embedding model. By combining the understanding ability of the large model, the direct word embedding vectorization processing of text data is transformed into information description of the text data before vectorization.

[0009] In order to achieve the above object, the present invention proposes a method for constructing a vector knowledge base based on a large model, comprising:

[0010] Text segmentation: segmenting a large amount of text into multiple small amount of text according to organizational relationships, wherein the multiple small amount of text is a plurality of segmented texts;

[0011] Description information extraction: extract the description information of each segmented text separately;

[0012] Key description concept extraction: extract the key description concepts of each segmented text separately;

[0013] Concept data dictionary output: All extracted key description concepts are filtered and merged, the relationships between key description concepts are sorted out according to the concept relationship rules, and the sorted relationships between key description concepts are output in the form of a data dictionary;

[0014] Vector knowledge base integration: convert the description information and data dictionary into corresponding vectors respectively, map and integrate the converted vectors, and save the integrated vectors into the vector database.

[0015] In one embodiment, in the text segmentation, the organizational relationship includes: topic, paragraph, chapter, or sentence.

[0016] In one embodiment, in the text segmentation, the organizational relationship also includes: title or word count.

[0017] In one embodiment, in the output of the concept data dictionary, filtering and merging includes filtering repeated key description concepts and merging similar key description concepts.

[0018] In one embodiment, in the concept data dictionary output, the concept relationship rules are inferred based on large amounts of text.

[0019] In one embodiment, in the concept data dictionary output, the concept relationship rules include logical relationship, causal relationship, correlation relationship, sequential relationship, and relevance.

[0020] In one embodiment, the description information extraction is performed in parallel with the key description concept extraction, or the description information extraction is performed after the key description concept extraction.

[0021] In one embodiment, the converting of the description information and the data dictionary into corresponding vectors respectively comprises: converting the text information into a vector representation in a high-dimensional space by using natural language processing technology, wherein the natural language processing technology includes word embedding, sentence embedding or document embedding.

[0022] In order to achieve the above object, the present invention further proposes a vector knowledge base construction system based on a large model, which is applied to the vector knowledge base construction method based on a large model as described above, comprising:

[0023] A text segmentation module is used to segment the input large data volume text into multiple small data volume texts according to the organizational relationship, wherein the multiple small data volume texts are multiple segmented texts;

[0024] An information description module is used to extract description information of each segmented text;

[0025] The concept extraction module is used to extract the key descriptive concepts of each segmented text;

[0026] A concept filtering and merging module is used to filter and merge all the extracted key description concepts, wherein the filtering and merging includes filtering repeated key description concepts and merging similar key description concepts;

[0027] The concept relationship arrangement module is used to arrange the relationship between each key description concept according to the concept relationship rules, and output the arranged relationship between each key description concept in the form of a data dictionary;

[0028] A large model module, used to cooperate with the information description module, the concept extraction module, the concept filtering and merging module and the concept relationship arrangement module to realize corresponding functions;

[0029] The word embedding module is used to convert the description information and data dictionary into corresponding vectors respectively;

[0030] The vector knowledge base integration module is used to map and integrate the converted vectors and save the integrated vectors into the vector database.

[0031] In one embodiment, the information description module, the concept extraction module, the concept filtering and merging module, and the concept relationship arrangement module implement corresponding functions through a large model module in the form of preset task prompt words.

[0032] The vector knowledge base construction method and system based on a large model of the present invention have the following beneficial effects:

[0033] 1. Compared with the single vectorization processing of the entire text, vectorization of information after large model information description processing can better maintain the information expressed therein. Highly refined vectorization processing can better express the main information therein and is less affected by the text vectorization method.

[0034] 2. Introducing concept representation in the process of building a local knowledge base will be more helpful for the retrieval and understanding of text information and the acquisition of related text information such as contextual logical causal correlation. Compared with a single vector similarity comparison on text, combining concept information can obtain higher quality content retrieval.

[0035] 3. Use the reasoning and understanding ability of the big model to uniformly extend and link the concepts extracted from a single paragraph of text information, so that the concepts can be combined with the specific content. This will play a more intuitive and effective role in linking the logical causal relationship between concepts, and further improve the level of the constructed local knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The figure is a flowchart of a method for constructing a vector knowledge base based on a large model according to an embodiment of the present invention.

[0037] Figure 2 The schematic diagram is a structural diagram of a large model-based vector knowledge base construction system according to an embodiment of the present invention.

[0038] Among them, 1. Text segmentation module; 2. Information description module; 3. Concept extraction module; 4. Concept filtering and merging module; 5. Concept relationship organization module; 6. Large model module; 7. Word embedding model module; and 8. Vector knowledge base integration module. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the invention and are not used to limit the invention.

[0040] The main innovation of this invention is to introduce a large model into the vectorization construction process of the local knowledge base. Considering that the current knowledge base vectorization method is limited to the classification of words and has limited ability to understand sentences, as well as the information loss caused by the single vectorization of large text data, the ability of the large model is used to create a knowledge base construction method that is more suitable for the vectorization of word embedding models.

[0041] The present invention proposes a method for constructing a vector knowledge base based on a large model, comprising:

[0042] Text segmentation: Segment a large amount of text into multiple small amounts of text according to organizational relationships, where the multiple small amounts of text are multiple segmented texts. The purpose of this step is to reduce the size of the processing unit and improve the efficiency of subsequent steps.

[0043] Descriptive information extraction: Extract descriptive information of each segmented text. This usually includes the topic, summary, key sentences, etc. of the text, which is used for subsequent understanding and query of the text content.

[0044] Key description concept extraction: Extract the key description concepts of each segmented text. These concepts are the core words or phrases in the text, representing the main content and features of the text.

[0045] Concept data dictionary output: All extracted key description concepts are filtered and merged. Filtering and merging includes filtering duplicate key description concepts and merging similar key description concepts to reduce redundancy and improve data consistency. The relationships between key description concepts are sorted out according to the concept relationship rules, and the sorted relationships between key description concepts are output in the form of a data dictionary. This data dictionary is an important part of the vector knowledge base.

[0046] Vector knowledge base integration: Convert the description information and data dictionary into corresponding vectors. This usually involves text vectorization techniques such as word embedding, sentence embedding or document embedding. The converted vectors are then mapped and integrated to ensure that the logical relationship between them is maintained in the vector space. The integrated vectors are saved to the vector database, which supports efficient vector query and retrieval operations.

[0047] This method combines the understanding ability of the large model to transform the direct word embedding vectorization processing of text data into information description of text data before vectorization. At the same time, when describing the information of text data, concepts in the text are extracted, and unified concept extension links are performed later. This method uses the understanding ability of the large model to solve the problems of word embedding vectorization by summarizing information and extracting concepts from segmented texts, so as to obtain a better organized knowledge vector library.

[0048] The following is a detailed description of some steps in the method for constructing a vector knowledge base based on a large model described above.

[0049] In the text segmentation step, the organizational relationship includes: theme, paragraph, chapter, sentence, and title or word count. In one embodiment, the large amount of text is first segmented according to the organizational relationship, with each chapter as a sub-document, and then further divided according to the rules such as title and word count. Wherein the paragraphs and specific character marks need to be further processed to ensure that the segmented text is complete. If a chapter has more text, it can be further divided into sub-documents in the same way.

[0050] In the step of outputting the concept data dictionary, the concept relationship rules are inferred based on the large amount of text. Specifically, the concept relationship rules include logical relationship, causal relationship, correlation relationship, sequential relationship, and relevance.

[0051] In addition, the two steps of extracting description information and extracting key description concepts can be performed in parallel, or the extraction of description information can be performed after the extraction of key description concepts. Parallel processing can speed up the processing, but attention should be paid to data synchronization and dependency.

[0052] In the step of vector knowledge base integration, the converted vectors are mapped for efficient storage and retrieval in the vector database. This may involve further processing of the vectors, such as dimensionality reduction and normalization, to improve query efficiency. This step is mainly about mapping between concepts and texts, ensuring that the correspondence between concepts to texts, texts to concepts, and concepts to concepts can be matched.

[0053] Finally, the mapped and integrated vectors and their metadata are saved in a specially designed vector database. The vector database is optimized for the storage and query of vector data, and can support efficient similarity search, cluster analysis and other operations. In addition, indexes can be established in the vector database to speed up the query. The selection and design of indexes depend on the specific application scenario and data characteristics. The query operation can also be optimized to ensure that relevant vectors and concepts can be quickly found even in massive data.

[0054] It should be noted that this process is not linear, and there may be iterations and feedback between the various steps. For example, in the process of text vectorization, it may be necessary to adjust the vectorization parameters and methods according to the performance and query requirements of the vector database. In the process of vector mapping and integration, it may also be necessary to adjust the mapping strategy and integration method according to the actual data distribution and query mode.

[0055] The present invention also proposes a vector knowledge base construction system based on a large model, which is applied to the vector knowledge base construction method based on the large model described above, and includes: a text segmentation module 1, an information description module 2, a concept extraction module 3, a concept filtering and merging module 4, a concept relationship arrangement module 5, a large model module 6, a word embedding model module 7 and a vector knowledge base integration module 8. The text segmentation module 1 is used to segment the input large data volume text into multiple small data volume texts according to the organizational relationship, wherein the multiple small data volume texts are multiple segmented texts, and the text segmentation module 1 is based on a rule algorithm to implement the knowledge text data from the large data volume text into multiple small data volume texts. The information description module 2 is used to extract the description information of each segmented text. The concept extraction module 3 is used to extract the key description concept of each segmented text. The concept filtering and merging module 4 is used to filter and merge all the extracted key description concepts, and the filtering and merging includes filtering repeated key description concepts and merging similar key description concepts. The concept relationship arrangement module 5 is used to arrange the relationship between each key description concept according to the concept relationship rule, and output the arranged relationship between each key description concept in the form of a data dictionary. The large model module 6 is used to cooperate with the information description module 2, the concept extraction module 3, the concept filtering and merging module 4 and the concept relationship arrangement module 5 to realize the corresponding functions. The word embedding model module 7 is used to convert the description information and the data dictionary into corresponding vectors respectively. The vector knowledge base integration module 8 is used to map and integrate the converted vectors, and finally save the integrated vectors to the vector database.

[0056] The large model module 6 is a general large language reasoning model. The information description module 2, the concept extraction module 3, the concept filtering and merging module 4 and the concept relationship arrangement module 5 implement corresponding functions through the large model module in the form of preset task prompt words. The details are as follows:

[0057] The input of the information description module 2 is the segmented text, which is packaged and output to the large model module 6 in the form of preset task prompt words. The output after reasoning by the large model module 6 is the refined information description of the text. The task prompt words can be designed as follows: "As an assistant with the ability to summarize and generalize information, you are very good at summarizing and refining the information of the text. There will be a paragraph of text next. Please briefly summarize one or a few sentences to describe what information this paragraph mainly explains."

[0058] The concept extraction module 3 is the same as the information description module 2. Its task is to extract the key descriptive concepts in the text. The task prompt words can be designed as follows: "As an assistant who is sensitive to conceptual information, you are very concerned about what concepts are contained in a text. Next, there will be a text. Please output what concepts are contained in it. These related concepts should be highly related to the main information expressed in this text."

[0059] Concept filtering and merging module 4, its task is to filter and merge the key descriptive concepts extracted from multiple texts to ensure that the concept information before and after is consistent and reasonable. Filtering and merging includes filtering repeated key descriptive concepts and merging similar key descriptive concepts. The task prompt can be designed as follows: "There are many similar concept expressions. As an expert in the field of linguistics, you are very sensitive to the similarity between concepts. Please integrate similar concepts into the same concept and output their corresponding relationship."

[0060] The task of the concept relationship rule sorting module 5 is to sort out the relationships between different concepts in the knowledge text, and to summarize and sort out concepts with logical relationships, causal relationships, or correlations. The task prompt can be designed as follows: "Please list the causal relationships, correlations, and sequential relationships between these different concepts. These relationships are inferred based on the following related texts. Sort out as many correlations as possible, and finally output them in the form of a data dictionary."

[0061] The above four modules all need to be used in conjunction with the large model module 6. The process may involve optimization and adaptation of prompt words, which are also some necessary debugging steps for application development based on the large language model.

[0062] The word embedding model module 7 is relatively simple, calling the model key text to be newly processed into a high-dimensional vector. The text information can be converted into a vector representation in a high-dimensional space by using natural language processing (NLP) technology, such as word embedding, sentence embedding or document embedding. In this embodiment, the text information (including descriptive information and concepts in the data dictionary) is converted into a vector representation in a high-dimensional space by word embedding. These vectors can capture the semantic information in the text, so that similar texts have a close distance in the vector space.

[0063] The vector knowledge base integration module 8 maps the converted vectors for efficient storage and retrieval in the vector database. This module matches the description information of the segmented text with the key description concepts extracted during the process and their extended links.

[0064] Example

[0065] The following is a description information extraction result and a key description concept extraction result of the segmented text using a large model-based vector knowledge base construction method and system according to an embodiment of the present invention.

[0066] The segmented text:

[0067] Child seats Topic Introduction Warning labels in the car vary slightly from car to car This section contains information on the following topics – General information on travelling with children in the car – Using child seats in the rear seat – Securing child seats with lower anchor points ISOFIX – Securing child seats with top fastening straps Warning ● All occupants, especially children, must always sit in the correct position and wear seat belts when driving. ● Do not share a seat belt with infants and young children in your arms, otherwise it is very likely to kill the infant or young child in a crash ● Do not let children move around freely in the car or stand in the car or kneel on the seat when driving. In a crash, the child may be thrown out of position and seriously injured, and may also injure other occupants in the car. ● If the child is not sitting in the correct position while the car is driving, it is very likely to be injured in emergency braking or an accident, especially when driving with children in the front seat. The situation is more serious when the airbag is triggered, which may kill the child ● Choosing a suitable child seat can effectively protect the child ● Make sure there are no hard or sharp objects, such as toys, on the child seat. These objects may injure the child in an accident. ● Children in the car should be supervised by an adult and should never be left alone in the car. ●Due to local climate conditions, the interior of the vehicle may become extremely cold or hot. This situation can be fatal to both people and animals in the vehicle. ●Children under 1.5 meters in height must not use regular seat belts without a child restraint system. Otherwise, the child's abdomen and neck may be injured in the event of emergency braking or an accident. ●Do not twist the seat belt or allow it to get stuck or rub against sharp edges. ●Even in a minor collision or sharp turn, the occupant may be injured if the seat belt is not worn properly. ●Only when the seat belt is worn correctly can it fully play its protective role. ●Do not allow two children to share a child seat. Warning: In the event of sudden emergency braking or emergency driving operations or in the event of an accident, an unsecured, unused child seat may be thrown into the vehicle and cause injury. ●Unused child seats must be securely fixed or securely stored in the luggage compartment during driving. ●Children must not sit in the side deployment area. Children who are sitting incorrectly and not using a child safety seat may be injured by the side airbag. ●Do not use a rear-facing child restraint system on a seat that is protected by an activated front airbag. Replace child seats that have been subjected to heavy loads after an accident. This is because invisible damage may have occurred. General information for travelling with children in the vehicle Note the information on page 49 at the beginning of this section. Various regulations and laws generally take precedence over the information in this owner's manual. There are different standards and regulations for the use of child seats and how they should be installed. For example, in some countries it may be prohibited to use child seats in certain seats in the vehicle. The same physical laws that apply to vehicles in the event of a collision or other type of accident apply to children. However, unlike adults and young people, children's muscles and joints are not yet fully developed. The risk of injury in an accident is much higher for children than for adults. Because children's bodies are not fully developed, it is recommended that they be used with a restraint system that is specifically adapted to their size, weight and physique. In many countries, it is legally required to use approved child restraint systems for infants and young children.Only use child seats that are approved and permitted for the car in question. If in doubt, consult your dealer. Checklist When travelling with children in the car, observe the country-specific legal regulations. It is recommended that children under 12 years of age always travel in the rear seat.

[0068] Descriptive information: How to install and use the child seat, precautions and risks involved.

[0069] Key descriptive concepts: child seat, installation, risk.

[0070] The vector knowledge base construction method and system based on a large model of the present invention have the following beneficial effects:

[0071] 1. Compared with the single vectorization processing of the entire text, vectorization of information after large model information description processing can better maintain the information expressed therein. Highly refined vectorization processing can better express the main information therein and is less affected by the text vectorization method.

[0072] 2. Introducing concept representation in the process of building a local knowledge base will be more helpful for the retrieval and understanding of text information and the acquisition of related text information such as contextual logical causal correlation. Compared with a single vector similarity comparison on text, combining concept information can obtain higher quality content retrieval.

[0073] 3. Use the reasoning and understanding ability of the big model to uniformly extend and link the concepts extracted from a single paragraph of text information, so that the concepts can be combined with the specific content. This will play a more intuitive and effective role in linking the logical causal relationship between concepts, and further improve the level of the constructed local knowledge base.

[0074] It should be understood that the entire vector knowledge base construction method and system based on the big model can not only target traditional text data, but also perform similar information description and conceptual extraction processing on image data through the visual big model. Finally, a local knowledge base combining concept and information description is constructed.

[0075] The above-described embodiments are only further explanations of the present invention, and are not intended to limit the present invention in other forms. The present invention may also have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding modifications and changes according to the present invention, but these corresponding modifications and changes should all fall within the protection scope of the present invention.

Claims

1. A method for constructing a vector knowledge base based on a large model, characterized in that: include: Text segmentation: segmenting a large amount of text into multiple small amount of text according to organizational relationships, wherein the multiple small amount of text is a plurality of segmented texts; Description information extraction: extract the description information of each segmented text separately; Key description concept extraction: extract the key description concepts of each segmented text separately; Concept data dictionary output: All extracted key description concepts are filtered and merged, the relationships between key description concepts are sorted out according to the concept relationship rules, and the sorted relationships between key description concepts are output in the form of a data dictionary; Vector knowledge base integration: convert the description information and data dictionary into corresponding vectors respectively, map and integrate the converted vectors, and save the integrated vectors into the vector database.

2. The method for constructing a vector knowledge base based on a large model according to claim 1, characterized in that: In the text segmentation, the organizational relationship includes: topic, paragraph, chapter, or sentence.

3. The method for constructing a vector knowledge base based on a large model according to claim 2, characterized in that: In the text segmentation, the organizational relationship also includes: title or word count.

4. The method for constructing a vector knowledge base based on a large model according to claim 1, characterized in that: In the output of the concept data dictionary, filtering and merging includes filtering repeated key description concepts and merging similar key description concepts.

5. The method for constructing a vector knowledge base based on a large model according to claim 1, characterized in that: In the conceptual data dictionary output, the conceptual relationship rules are inferred based on large amounts of text.

6. The method for constructing a vector knowledge base based on a large model according to claim 1, characterized in that: In the output of the concept data dictionary, the concept relationship rules include logical relationship, causal relationship, correlation relationship, sequential relationship, and relevance.

7. The method for constructing a vector knowledge base based on a large model according to claim 1, characterized in that: The description information extraction is performed in parallel with the key description concept extraction, or the description information extraction is performed after the key description concept extraction.

8. The method for constructing a vector knowledge base based on a large model according to claim 1, characterized in that: The converting of the description information and the data dictionary into corresponding vectors respectively comprises: converting the text information into a vector representation in a high-dimensional space by using a natural language processing technique, wherein the natural language processing technique comprises word embedding, sentence embedding or document embedding.

9. A vector knowledge base construction system based on a large model, characterized in that: The method for constructing a vector knowledge base based on a large model as claimed in any one of claims 1 to 8 comprises: A text segmentation module is used to segment the input large data volume text into multiple small data volume texts according to the organizational relationship, wherein the multiple small data volume texts are multiple segmented texts; An information description module is used to extract description information of each segmented text; The concept extraction module is used to extract the key descriptive concepts of each segmented text; A concept filtering and merging module is used to filter and merge all the extracted key description concepts, wherein the filtering and merging includes filtering repeated key description concepts and merging similar key description concepts; The concept relationship arrangement module is used to arrange the relationship between each key description concept according to the concept relationship rules, and output the arranged relationship between each key description concept in the form of a data dictionary; A large model module, used to cooperate with the information description module, the concept extraction module, the concept filtering and merging module and the concept relationship arrangement module to realize corresponding functions; The word embedding module is used to convert the description information and data dictionary into corresponding vectors respectively; The vector knowledge base integration module is used to map and integrate the converted vectors and save the integrated vectors into the vector database.

10. The vector knowledge base construction system based on a large model according to claim 9 is characterized in that: The information description module, concept extraction module, concept filtering and merging module and concept relationship arrangement module realize corresponding functions through the large model module in the form of preset task prompt words.

Images