Hybrid retrieval knowledge base construction system and method based on keyword library and vector

Abstract

The application provides a keyword library and vector-based hybrid retrieval knowledge base construction system and method; the system comprises a facade module, a timing module, a knowledge base creation module, a knowledge query module, a keyword module and a vector module; the method comprises the following sub-steps: S1, creating a knowledge base and cutting uploaded knowledge documents into text blocks; S2, creating a keyword library and saving obtained keywords in the keyword library; S3, updating keywords of the knowledge documents by using the content in the keyword library; S4, hybrid keyword and vector query; the method reduces the need for data uploading by localizing keyword and vector calculation, thereby effectively reducing the risk of data leakage and enhancing the applicability of the system in commercial products; by combining keyword retrieval and vector retrieval technologies and by keyword and vector joint query, the accuracy and recall rate of knowledge retrieval are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a system and method for constructing a hybrid retrieval knowledge base based on keyword databases and vectors. Background Technology

[0002] With the development of artificial intelligence technology, large language model LLM has been widely used in various industries, from natural language processing (NLP) to intelligent customer service and personalized recommendation systems. The application of large language model LLM is constantly expanding. Although large language model LLM has powerful language understanding and generation capabilities, it still has some limitations, including the illusion problem, knowledge update lag, and lack of domain knowledge.

[0003] Using knowledge bases can overcome the limitations of large language models. The core of this approach is to introduce an external knowledge base on top of the large language model (LLM) to dynamically enhance the LLM's answering capabilities. This technical framework that combines retrieval and generation capabilities is called Retrieval Enhanced Generation (RAG).

[0004] Although current knowledge-based retrieval augmentation generation (RAG) technology has improved the capabilities of large language models to some extent, it still has the following technical shortcomings:

[0005] Data privacy risks of commercial products: The existing knowledge base contains commercial and open source products. During the implementation of commercial products, it is usually necessary to upload document fragments of enterprises or users to the service center for processing. While this is convenient and efficient, it also brings the risk of data leakage.

[0006] Open source products suffer from insufficient recall and precision: Open source products rely solely on vector similarity for retrieval, resulting in insufficient recall and precision. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a hybrid retrieval knowledge base construction system and method based on keyword databases and vectors.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A hybrid retrieval knowledge base construction system based on keyword database and vectors includes a facade module, a timing module, a knowledge base creation module, a knowledge query module, a keyword module, and a vector module;

[0010] The facade module serves as the entry point for interaction with the outside world, providing interfaces for knowledge base creation, document upload, etc.

[0011] The timing module is used to update the keyword database at regular intervals and reconstruct the keywords of the text based on the database content;

[0012] The knowledge base creation module is used to create a knowledge base and upload and add knowledge documents to the knowledge base from time to time.

[0013] The knowledge query module is used to dynamically adjust the weights of keywords and vectors, query knowledge, and sort it according to the comprehensive score;

[0014] The keyword module is used to extract and update keywords;

[0015] The vector module is used to vectorize each text block in the knowledge base.

[0016] Specifically, the method for constructing a hybrid retrieval knowledge base based on keyword databases and vectors includes the following sub-steps:

[0017] S1: Create a knowledge base and segment the uploaded knowledge documents into text blocks;

[0018] The knowledge base creation module creates a knowledge base based on the knowledge base creation interface, and uploads and adds knowledge documents and user-submitted question texts to the created knowledge base from time to time through the document upload interface.

[0019] The knowledge document is divided into text blocks. The specific segmentation method is as follows:

[0020] Read the knowledge document and segment its content according to its type. The knowledge document types include structured types (such as CSV, Excel, etc.) and unstructured types (such as TXT, PDF, Markdown, Word, etc.).

[0021] For structured knowledge documents, text is segmented into text blocks based on the lines within the document.

[0022] For unstructured knowledge documents, the text is segmented into text blocks based on paragraphs within the document.

[0023] If the number of characters in the segmented text blocks is still greater than the pre-set text block character threshold, then use natural language processing tools (such as spaCy or NLTK) to continue segmenting the text blocks by sentences until all text blocks are smaller than the text block character threshold.

[0024] All the segmented text blocks constitute a text block list. Each text block in the text block list corresponds to a knowledge record. Each knowledge record includes a knowledge record ID, the text content of the text block, and the data source of the text content (file name, page number, line number information), etc.

[0025] The knowledge record ID is a unique identifier;

[0026] The vector module passes the segmented text blocks to the text embedding model, which converts the text blocks into vectors, standardizes the vectors (scales them to unit vectors with a modulus of 1), and saves them to the vector library.

[0027] Furthermore, in the knowledge base, a status field is set for each text block and the question text extracted by the user. For newly added text blocks that have not been subject to keyword extraction or question text extracted by the user, the status field is initialized to "not extracted"; for text blocks that have been subject to keyword extraction or question text extracted by the user, the status field is updated to "extracted".

[0028] S2: Create a keyword library and save the acquired keywords to the keyword library;

[0029] Includes the following sub-steps:

[0030] S21: Save the user-specified keywords to the keyword library;

[0031] Users set unique keywords for the current knowledge base, and the keywords are saved to the keyword library after being deduplicated.

[0032] For example, the keywords in the user-defined knowledge base are "DPF T box vehicle idling fault code clearing water temperature tire pressure carbon load limit torque limit speed regeneration", with the keywords separated by spaces;

[0033] S22: Save the keywords generated by the large model to the keyword library;

[0034] Based on the user's pre-defined summary of the purpose of the knowledge base, a large model is used to summarize keywords, and the summarized keywords are saved to the keyword library;

[0035] S23: Save the keywords extracted from the knowledge documents to the keyword library;

[0036] The timing module sets the scheduled tasks and trigger times for the knowledge base (e.g., 1 a.m. every day). The content of the scheduled task is to extract keywords from the text blocks of the knowledge document.

[0037] The scheduled task is executed at the trigger time. The specific execution method is as follows:

[0038] The keyword module retrieves text blocks newly added to the knowledge base within a set time period (e.g., the previous day) from the knowledge base using a query statement; and determines whether there are any text blocks that have not been extracted by keywords based on the status field. If there are no text blocks that have not been extracted by keywords, this execution step ends and proceeds to step S24.

[0039] If there are text blocks for which keywords have not been extracted, the corresponding keywords are extracted from the Chinese word segmentation library Jieba in Python. Keywords that do not exist in the keyword library are then filtered out using a SELECT query.

[0040] Save the selected keywords to the corresponding knowledge records in the knowledge base;

[0041] After deduplication of the extracted keywords, the frequency of the keywords is calculated from the keywords that do not exist in the keyword library using the collections.Counter component, and the frequency of the keywords is sorted to obtain the keywords with a frequency greater than 3 and ranked in the top ten. The large language model is called to extract professional keywords from the keywords with a frequency greater than 3 and ranked in the top 10 and save them to the keyword library.

[0042] S24: Save the keywords extracted from user questions to the keyword library;

[0043] The timed module sets the scheduled tasks and trigger times for the knowledge base (e.g., 2 AM every day). The content of the scheduled tasks is to extract keywords from the text of questions raised by users.

[0044] The scheduled task is executed at the trigger time. The specific execution method is as follows:

[0045] The keyword module retrieves the text of questions raised by users who have newly joined the knowledge base within a set time period (e.g., the previous day) from the knowledge base through a query statement, and determines whether there are any question texts that have not been extracted by keywords based on the status field. If there are no question texts that have not been extracted by keywords, this execution step ends and proceeds to step S3.

[0046] If there are text blocks for which no keywords have been extracted, the Python Chinese word segmentation library Jieba is called to extract the corresponding keywords from the text for which no keywords have been extracted. After deduplication of the extracted keywords, the frequency of the extracted keywords is calculated using the collections.Counter component, and keywords that do not exist in the keyword library are filtered out using a SELECT query.

[0047] The selected keywords are sorted by word frequency to obtain the top 5 keywords with a word frequency greater than 3. The large language model is then used to extract professional keywords from these keywords and save them to the keyword library.

[0048] S3: Update the keywords in the knowledge document using the keyword library;

[0049] The timed module sets the scheduled tasks and trigger times for the knowledge base (e.g., 3 AM every day). The content of the scheduled task is to update the knowledge documents with keywords using the content in the keyword library.

[0050] The scheduled task is executed at the trigger time. The specific execution method is as follows:

[0051] The keyword module retrieves text blocks newly added to the knowledge base within a set time period (e.g., the previous day) from the knowledge base using query statements; it iterates through each text block, matches the keywords in the keyword library with the text block, and obtains the keywords in the text block.

[0052] The keywords in each text block are merged with the keywords of the corresponding knowledge records in the knowledge base, deduplicated, and then saved to the keyword database and the corresponding knowledge records to realize the keyword update of the knowledge document.

[0053] S4: Perform mixed knowledge queries using keywords and vectors;

[0054] Includes the following sub-steps:

[0055] S41: Calculate the vector query weight and keyword query weight based on the actual question text submitted by the user;

[0056] The length component is used to obtain the number of characters in the user's question, and the vector query weight and keyword weight are calculated based on the number of characters using the following formula;

[0057] ;

[0058] Where weight_vector is the vector query weight, and weight_keyword is the keyword weight;

[0059] S42: Filter out keywords that exist in the keyword database from the question text;

[0060] The filter component iterates through the keyword library and matches it with the question text to filter out the keywords contained in the question text that exist in the keyword library, which are then used as the question keywords for this question text.

[0061] S43: Obtain the knowledge records of the top five keyword matching scores;

[0062] Traverse each knowledge record in the knowledge base and filter out the knowledge records that contain any keyword obtained in step S32;

[0063] Each retrieved knowledge record is scored based on keyword matching, using the following formula:

[0064] ;

[0065] Where score_keyword is the keyword matching score;

[0066] The sorted function is used to sort all the keyword matching scores in descending order, and the top five knowledge records with the highest scores are obtained.

[0067] S44: Obtain the knowledge records of the top five vector distance scores;

[0068] The vector module inputs the question text into the text embedding model, performs vector transformation, and then standardizes the transformed vector by scaling it to a unit vector with a modulus of 1 to obtain the question vector.

[0069] The vector distances between the problem vector and the vectors in the vector library are calculated sequentially using the Euclidean distance formula.

[0070] The vector distance score is calculated using the following formula;

[0071] score_vector = 1 - vector distance;

[0072] Where score_vector represents the vector distance score;

[0073] The sorted function is used to sort all the vector distance scores in descending order, and the top five knowledge records with the highest scores are obtained.

[0074] S45: Calculate the comprehensive score of the knowledge record based on the vector query weight and keyword query weight;

[0075] Merge the knowledge records obtained in steps S43 and S44, and perform deduplication on the merged knowledge records according to the knowledge record ID;

[0076] The knowledge query module calculates the comprehensive score of the deduplicated knowledge records based on the vector query weights and keyword query weights obtained in step S41.

[0077] ;

[0078] Among them, score is the overall score of each knowledge record;

[0079] Sort the overall scores in descending order and return the top five knowledge records, which are the knowledge records corresponding to this problem.

[0080] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0081] This method reduces the need for data uploads by using localized keywords and vector calculations, thereby effectively reducing the risk of data leakage and enhancing the applicability of the system in commercial products.

[0082] By combining keyword retrieval and vector retrieval technologies, and through joint keyword and vector queries, it is possible to more accurately match questions and knowledge documents, improve the accuracy and relevance of retrieval, effectively improve the accuracy and recall rate of knowledge retrieval, and overcome the limitations of traditional technologies based on single vector similarity retrieval methods.

[0083] By employing a timed module and a keyword module, keywords are extracted periodically from knowledge documents and user questions. The keyword database is then updated in real time using natural language processing tools. This eliminates the need for manual intervention or regular updates, reducing maintenance costs, improving the timeliness of the keyword and knowledge databases, ensuring the system responds promptly to the latest knowledge, and enhancing its intelligence level. Attached Figure Description

[0084] Figure 1 This is a flowchart illustrating the steps of the method for constructing a hybrid retrieval knowledge base based on a keyword database and vectors according to the present invention.

[0085] Figure 2 This is a module distribution diagram of the hybrid retrieval knowledge base construction system based on keyword database and vectors of the present invention. Detailed Implementation

[0086] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0087] A hybrid retrieval knowledge base construction system based on keyword database and vectors includes a facade module, a timing module, a knowledge base creation module, a knowledge query module, a keyword module, and a vector module;

[0088] The facade module serves as the entry point for interaction with the outside world, providing interfaces for knowledge base creation, document upload, etc.

[0089] The timing module is used to update the keyword database at regular intervals and reconstruct the keywords of the text based on the database content;

[0090] The knowledge base creation module is used to create a knowledge base and upload and add knowledge documents to the knowledge base from time to time.

[0091] The knowledge query module is used to dynamically adjust the weights of keywords and vectors, query knowledge, and sort it according to the comprehensive score;

[0092] The keyword module is used to extract and update keywords;

[0093] The vector module is used to vectorize each text block in the knowledge base.

[0094] like Figure 1 , 2As shown, the method for constructing a hybrid retrieval knowledge base based on keyword databases and vectors includes the following sub-steps:

[0095] S1: Create a knowledge base and segment the uploaded knowledge documents into text blocks;

[0096] The knowledge base creation module creates a knowledge base based on the knowledge base creation interface, and uploads and adds knowledge documents and user-submitted question texts to the created knowledge base from time to time through the document upload interface.

[0097] The knowledge document is divided into text blocks. The specific segmentation method is as follows:

[0098] Read the knowledge document and segment its content according to its type. The knowledge document types include structured types (such as CSV, Excel, etc.) and unstructured types (such as TXT, PDF, Markdown, Word, etc.).

[0099] For structured knowledge documents, text is segmented into text blocks based on the lines within the document.

[0100] For unstructured knowledge documents, the text is segmented into text blocks based on paragraphs within the document.

[0101] If the number of characters in the segmented text blocks is still greater than the pre-set text block character threshold, then use natural language processing tools (such as spaCy or NLTK) to continue segmenting the text blocks by sentences until all text blocks are smaller than the text block character threshold.

[0102] All the segmented text blocks constitute a text block list. Each text block in the text block list corresponds to a knowledge record. Each knowledge record includes a knowledge record ID, the text content of the text block, and the data source of the text content (file name, page number, line number information), etc.

[0103] The knowledge record ID is a unique identifier;

[0104] The vector module passes the segmented text blocks to the text embedding model, which converts the text blocks into vectors, standardizes the vectors (scales them to unit vectors with a modulus of 1), and saves them to the vector library.

[0105] Furthermore, in the knowledge base, a status field is set for each text block and the question text extracted by the user. For newly added text blocks that have not been subject to keyword extraction or question text extracted by the user, the status field is initialized to "not extracted"; for text blocks that have been subject to keyword extraction or question text extracted by the user, the status field is updated to "extracted".

[0106] By segmenting knowledge documents at multiple levels, including differentiated processing of structured and unstructured documents, it can better support knowledge documents from various sources and provide more flexible and extensive application scenarios. Combined with text vectorization technology, each text block is converted into a vector for retrieval, improving retrieval accuracy and solving the problems of information being scattered or too verbose in large documents.

[0107] By setting a status field for each knowledge record to mark whether the document has undergone keyword extraction and processing, the system can ensure that it can track the processing progress of the document in real time, avoid omissions and duplicate processing, and improve the intelligence level of the system.

[0108] S2: Create a keyword library and save the acquired keywords to the keyword library;

[0109] Includes the following sub-steps:

[0110] S21: Save the user-specified keywords to the keyword library;

[0111] Users set unique keywords for the current knowledge base, and the keywords are saved to the keyword library after being deduplicated.

[0112] For example, the keywords in the user-defined knowledge base are "DPF T box vehicle idling fault code clearing water temperature tire pressure carbon load limit torque limit speed regeneration", with the keywords separated by spaces;

[0113] S22: Save the keywords generated by the large model to the keyword library;

[0114] Based on the user's pre-defined summary of the purpose of the knowledge base, a large model is used to summarize keywords, and the summarized keywords are saved to the keyword library;

[0115] S23: Save the keywords extracted from the knowledge documents to the keyword library;

[0116] The timing module sets the scheduled tasks and trigger times for the knowledge base (e.g., 1 a.m. every day). The content of the scheduled task is to extract keywords from the text blocks of the knowledge document.

[0117] The scheduled task is executed at the trigger time. The specific execution method is as follows:

[0118] The keyword module retrieves text blocks newly added to the knowledge base within a set time period (e.g., the previous day) from the knowledge base using a query statement; and determines whether there are any text blocks that have not been extracted by keywords based on the status field. If there are no text blocks that have not been extracted by keywords, this execution step ends and proceeds to step S24.

[0119] If there are text blocks for which keywords have not been extracted, the corresponding keywords are extracted from the Chinese word segmentation library Jieba in Python. Keywords that do not exist in the keyword library are then filtered out using a SELECT query.

[0120] Save the selected keywords to the corresponding knowledge records in the knowledge base;

[0121] After deduplication of the extracted keywords, the frequency of the keywords is calculated from the keywords that do not exist in the keyword library using the collections.Counter component, and the frequency of the keywords is sorted to obtain the keywords with a frequency greater than 3 and ranked in the top ten. The large language model is called to extract professional keywords from the keywords with a frequency greater than 3 and ranked in the top 10 and save them to the keyword library.

[0122] S24: Save the keywords extracted from user questions to the keyword library;

[0123] The timed module sets the scheduled tasks and trigger times for the knowledge base (e.g., 2 AM every day). The content of the scheduled tasks is to extract keywords from the text of questions raised by users.

[0124] The scheduled task is executed at the trigger time. The specific execution method is as follows:

[0125] The keyword module retrieves the text of questions raised by users who have newly joined the knowledge base within a set time period (e.g., the previous day) from the knowledge base through a query statement, and determines whether there are any question texts that have not been extracted by keywords based on the status field. If there are no question texts that have not been extracted by keywords, this execution step ends and proceeds to step S3.

[0126] If there are text blocks for which no keywords have been extracted, the Python Chinese word segmentation library Jieba is called to extract the corresponding keywords from the text for which no keywords have been extracted. After deduplication of the extracted keywords, the frequency of the extracted keywords is calculated using the collections.Counter component, and keywords that do not exist in the keyword library are filtered out using a SELECT query.

[0127] The selected keywords are sorted by word frequency to obtain the top 5 keywords with a word frequency greater than 3. The large language model is then used to extract professional keywords from these keywords and save them to the keyword library.

[0128] Keywords are extracted using a large language model and Chinese word segmentation tools. A keyword deduplication mechanism ensures the accuracy of the vocabulary in the keyword database and avoids redundancy.

[0129] S3: Update the keywords in the knowledge document using the keyword library;

[0130] The timed module sets the scheduled tasks and trigger times for the knowledge base (e.g., 3 AM every day). The content of the scheduled task is to update the knowledge documents with keywords using the content in the keyword library.

[0131] The scheduled task is executed at the trigger time. The specific execution method is as follows:

[0132] The keyword module retrieves text blocks newly added to the knowledge base within a set time period (e.g., the previous day) from the knowledge base using query statements; it iterates through each text block, matches the keywords in the keyword library with the text block, and obtains the keywords in the text block.

[0133] The keywords in each text block are merged with the keywords of the corresponding knowledge records in the knowledge base, deduplicated, and then saved to the keyword database and the corresponding knowledge records to realize the keyword update of the knowledge document.

[0134] S4: Perform mixed knowledge queries using keywords and vectors;

[0135] Includes the following sub-steps:

[0136] S41: Calculate the vector query weight and keyword query weight based on the actual question text submitted by the user;

[0137] The length component is used to obtain the number of characters in the user's question, and the vector query weight and keyword weight are calculated based on the number of characters using the following formula;

[0138] ;

[0139] Where weight_vector is the vector query weight, and weight_keyword is the keyword weight;

[0140] S42: Filter out keywords that exist in the keyword database from the question text;

[0141] The filter component iterates through the keyword library and matches it with the question text to filter out the keywords contained in the question text that exist in the keyword library, which are then used as the question keywords for this question text.

[0142] S43: Obtain the knowledge records of the top five keyword matching scores;

[0143] Traverse each knowledge record in the knowledge base and filter out the knowledge records that contain any keyword obtained in step S32;

[0144] Each retrieved knowledge record is scored based on keyword matching, using the following formula:

[0145] ;

[0146] Where score_keyword is the keyword matching score;

[0147] The sorted function is used to sort all the keyword matching scores in descending order, and the top five knowledge records with the highest scores are obtained.

[0148] S44: Obtain the knowledge records of the top five vector distance scores;

[0149] The vector module inputs the question text into the text embedding model, performs vector transformation, and then standardizes the transformed vector by scaling it to a unit vector with a modulus of 1 to obtain the question vector.

[0150] The vector distances between the problem vector and the vectors in the vector library are calculated sequentially using the Euclidean distance formula.

[0151] The vector distance score is calculated using the following formula;

[0152] score_vector = 1 - vector distance;

[0153] Where score_vector represents the vector distance score;

[0154] The sorted function is used to sort all the vector distance scores in descending order, and the top five knowledge records with the highest scores are obtained.

[0155] S45: Calculate the comprehensive score of the knowledge record based on the vector query weight and keyword query weight;

[0156] Merge the knowledge records obtained in steps S43 and S44, and perform deduplication on the merged knowledge records according to the knowledge record ID;

[0157] The knowledge query module calculates the comprehensive score of the deduplicated knowledge records based on the vector query weights and keyword query weights obtained in step S41.

[0158] ;

[0159] Among them, score is the overall score of each knowledge record;

[0160] Sort the overall scores in descending order and return the top five knowledge records, which are the knowledge records corresponding to this problem.

[0161] By comprehensively calculating keyword matching scores and vector distance scores, the knowledge records are sorted, taking into account both keyword matching degree and semantic similarity, ensuring that the returned knowledge records are the most relevant and providing accurate search results.

[0162] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. A method for constructing a hybrid retrieval knowledge base based on a keyword database and vectors, characterized in that: Includes the following steps: S1: Create a knowledge base and segment the uploaded knowledge documents into text blocks; The knowledge base creation module creates a knowledge base based on the knowledge base creation interface, and uploads and adds knowledge documents and user-submitted question texts to the created knowledge base from time to time through the document upload interface. The knowledge document is divided into text blocks, and all the divided text blocks form a text block list. Each text block in the text block list corresponds to a knowledge record. The vector module passes the segmented text blocks to the text embedding model, which converts the text blocks into vectors, standardizes the vectors (i.e., scales them to unit vectors with a modulus of 1), and saves them to the vector library. In the knowledge base, a status field is set for each text block and the question text extracted by the user. The status field corresponds to two states: keyword extraction has been performed and keyword extraction has not been performed. S2: Create a keyword library and save the acquired keywords to the keyword library; S21: Save the user-specified keywords to the keyword library; S22: Save the keywords generated by the large model to the keyword library; S23: Save the keywords extracted from the knowledge documents to the keyword library; S24: Save the keywords extracted from user questions to the keyword library; S3: Update the keywords in the knowledge document using the keyword library; The timing module sets the scheduled tasks and trigger times for the knowledge base, and executes the scheduled tasks at the trigger times; the content of the scheduled tasks is to update the keywords of the knowledge documents using the content in the keyword library; S4: Perform mixed knowledge queries using keywords and vectors; S41: Calculate the vector query weight and keyword query weight based on the actual question text submitted by the user; S42: Filter out keywords that exist in the keyword database from the question text; S43: Obtain the knowledge records of the top N keyword matching scores; N is a positive integer; S44: Obtain the knowledge records of the top N vector distance scores; S45: Calculate the comprehensive score of the knowledge record based on the vector query weight and keyword query weight.

2. The method for constructing a hybrid retrieval knowledge base based on a keyword database and vectors as described in claim 1, characterized in that: In step S1, the specific method for dividing the knowledge document into text blocks is as follows: Read knowledge documents and segment the content of the knowledge documents according to their types, including structured and unstructured types; For structured knowledge documents, text is segmented into text blocks based on the lines within the document. For unstructured knowledge documents, the text is segmented into text blocks based on paragraphs within the document. If the number of characters in the segmented text block is still greater than the pre-set text block character threshold, then the natural language processing tool is used to continue segmenting the text block by sentence until all text blocks are smaller than the text block character threshold. All the segmented text blocks constitute a text block list. Each text block in the text block list corresponds to a knowledge record. Each knowledge record includes a knowledge record ID, the text content of the text block, and the data source of the text content. The knowledge record ID is a unique identifier; In the knowledge base, a status field is set for each text block and the question text extracted by the user. For newly added text blocks or question texts that have not been subject to keyword extraction, the status field is initialized to "not extracted"; for text blocks or question texts that have been subject to keyword extraction, the status field is updated to "extracted".

3. The method for constructing a hybrid retrieval knowledge base based on a keyword database and vectors as described in claim 1, characterized in that: The specific details of step S2 are as follows: S21: Save the user-specified keywords to the knowledge base; Users set unique keywords for the current knowledge base, and the keywords are saved to the keyword library after being deduplicated. S22: Save the keywords generated by the large model to the knowledge base; Based on the user's pre-defined summary of the purpose of the knowledge base, a large model is used to summarize keywords, and the summarized keywords are saved to the keyword library; S23: Save the keywords extracted from the knowledge document to the knowledge base; The timing module sets the scheduled tasks and trigger times for the knowledge base. The content of the scheduled tasks is to extract keywords from the text blocks of the knowledge documents. The scheduled task is executed at the trigger time. The specific execution method is as follows: The keyword module retrieves text blocks newly added to the knowledge base within a set time period from the knowledge base through query statements; and determines whether there are text blocks that have not been extracted by keywords based on the status field. If there are no text blocks that have not been extracted by keywords, this execution step ends and proceeds to step S24. If there are text blocks for which keywords have not been extracted, the corresponding keywords are extracted from the Chinese word segmentation library Jieba in Python. Keywords that do not exist in the keyword library are then filtered out using a SELECT query. Save the selected keywords to the corresponding knowledge records in the knowledge base; Pre-set word frequency requirements; After deduplication of the extracted keywords, the frequency of the keywords is calculated from the keywords that do not exist in the keyword library using the collections.Counter component, and the frequency of the keywords is sorted to obtain keywords that meet the set frequency requirements; the large language model is called to extract professional keywords from the keywords that meet the set frequency requirements and save them to the keyword library; S24: Save the keywords extracted from user questions to the keyword library; The timing module sets the scheduled tasks and trigger times for the knowledge base. The content of the scheduled tasks is to extract keywords from the text of questions raised by users. The scheduled task is executed at the trigger time. The specific execution method is as follows: The keyword module retrieves the text of questions raised by users who have newly joined the knowledge base within a set time period from the knowledge base through a query statement, and determines whether there are any question texts that have not been extracted by keywords based on the status field. If there are no question texts that have not been extracted by keywords, this execution step ends and proceeds to step S3. If there are text blocks for which no keywords have been extracted, the Python Chinese word segmentation library Jieba is called to extract the corresponding keywords from the text for which no keywords have been extracted. After deduplication of the extracted keywords, the frequency of the extracted keywords is calculated using the collections.Counter component, and keywords that do not exist in the keyword library are filtered out using a SELECT query. The selected keywords are sorted by word frequency to obtain keywords that meet the set word frequency requirements; The large language model is invoked to extract specialized keywords from keywords that meet the set word frequency requirements and save them to the keyword library.

4. The method for constructing a hybrid retrieval knowledge base based on keyword database and vectors as described in claim 1, characterized in that: In step S3, a scheduled task is executed at the trigger time. The specific execution method is as follows: The keyword module retrieves text blocks newly added to the knowledge base within a set time period through query statements; it iterates through each text block, matches the keywords in the keyword library with the text block, and obtains the keywords in the text block. The keywords in each text block are merged with the keywords of the corresponding knowledge records in the knowledge base, deduplicated, and then saved to the keyword database and the corresponding knowledge records to realize the keyword update of the knowledge document.

5. The method for constructing a hybrid retrieval knowledge base based on a keyword database and vectors as described in claim 1, characterized in that: The specific details of step S4 are as follows: S41: Calculate the vector query weight and keyword query weight based on the actual question text submitted by the user; The length component is used to obtain the number of characters in the user's question text, and the vector query weight and keyword weight are calculated based on the number of characters using the following formula; ； Where weight_vector is the vector query weight, and weight_keyword is the keyword weight; S42: Filter out keywords that exist in the keyword database from the question text; The filter component iterates through the keyword library and matches it with the question text to filter out the keywords contained in the question text that exist in the keyword library, which are then used as the question keywords for this question text. S43: Obtain the knowledge records of the top N keyword matching scores; Traverse each knowledge record in the knowledge base and filter out the knowledge records that contain any keyword obtained in step S32; Each retrieved knowledge record is scored based on keyword matching, using the following formula: ； Where score_keyword is the keyword matching score; The sorted function is used to sort all the keyword matching scores in descending order, and the top N knowledge records are obtained. S44: Obtain the knowledge records of the top N vector distance scores; The vector module inputs the question text into the text embedding model, performs vector transformation, and then standardizes the transformed vector by scaling it to a unit vector with a modulus of 1 to obtain the question vector. The vector distances between the problem vector and the vectors in the vector library are calculated sequentially using the Euclidean distance formula. The vector distance score is calculated using the following formula; score_vector = 1 - vector distance; Where score_vector represents the vector distance score; The sorted function is used to sort all the vector distance scores in descending order, and the top N knowledge records with the highest scores are obtained. S45: Calculate the comprehensive score of the knowledge record based on the vector query weight and keyword query weight; Merge the knowledge records obtained in steps S43 and S44, and perform deduplication on the merged knowledge records according to the knowledge record ID; The knowledge query module calculates the comprehensive score of the deduplicated knowledge records based on the vector query weights and keyword query weights obtained in step S41. ； Among them, score is the overall score of each knowledge record; Sort the comprehensive scores in descending order and return the knowledge records with the highest scores (A), which are the knowledge records corresponding to this problem; where A is a positive integer that is actually set.

6. A hybrid retrieval knowledge base construction system based on keyword database and vector for implementing the method of any one of claims 1-5, characterized in that: It includes a facade module, a timing module, a knowledge base creation module, a knowledge query module, a keyword module, and a vector module; The facade module serves as the entry point for interaction with the outside world, providing interfaces for knowledge base creation and document upload. The timing module is used to update the keyword database at regular intervals and reconstruct the keywords of the text based on the database content; The knowledge base creation module is used to create a knowledge base and upload and add knowledge documents to the knowledge base from time to time. The knowledge query module is used to dynamically adjust the weights of keywords and vectors, query knowledge, and sort it according to the comprehensive score; The keyword module is used to extract and update keywords; The vector module is used to vectorize each text block in the knowledge base.

Images