Multi-embedding model compatible Chinese vector knowledge base construction and hybrid retrieval method, system, device and medium

Abstract

This invention provides a method, system, device, and medium for constructing and hybrid retrieval of a Chinese vector knowledge base compatible with multiple embedding models. The method utilizes embedding model configuration normalization and runtime dynamic construction, while adapting to both OpenAI's cloud-compatible interface and Ollam's local interface. Before knowledge base creation, dynamic vector dimension detection is performed to automatically generate a database table structure matching the current model. A dual-channel document representation is constructed, including the original text, Chinese word segmentation lexical channels, and embedded vector semantic channels, achieving comprehensive recall through vector similarity retrieval and full-text retrieval. Document-level fragment deletion, filtering, and reconstruction are achieved through file identification. This invention solves the problems of poor embedding model compatibility, insufficient Chinese precise retrieval capabilities, and high knowledge base maintenance costs in existing technologies, significantly improving the flexibility, retrieval accuracy, and maintainability of the Chinese vector knowledge base.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, vector databases, full-text retrieval, knowledge base construction, and intelligent question answering, and particularly to a method, system, device, and medium for constructing and hybridly retrieving Chinese vector knowledge bases compatible with multiple embedding models. Background Technology

[0002] Existing knowledge base question answering systems typically use embedding models to convert document content into vectors and store them in a vector database. When a user queries, the query text is converted into a query vector, and relevant fragments are returned based on vector similarity.

[0003] Existing vector knowledge base solutions have the following shortcomings in practical applications:

[0004] (1) Poor compatibility of embedded models: The vector dimensions output by different embedded models vary greatly. When using fixed-dimensional tables, dimension mismatch errors will occur when switching models, requiring the entire knowledge base to be rebuilt. At the same time, the interface forms of different embedded services are different. The OpenAI compatible interface and the Ollam native interface differ in authentication methods, address formats, and calling parameters, which increases the complexity of system integration.

[0005] (2) Insufficient Chinese search capability: Chinese text does not have natural space separation, and it is difficult to accurately match proper nouns, numbers, product models and other precise keywords by simply relying on vector similarity search, which is prone to missed detection; while simple full text search cannot identify semantically related content such as synonyms and hyponyms.

[0006] (3) High maintenance cost of knowledge base: Existing solutions mostly manage data at the granularity of fragments. When a document is updated or deleted, it is not possible to quickly locate and delete the corresponding fragment. The entire knowledge base can only be rebuilt, which is extremely inefficient, especially in large-scale knowledge base scenarios.

[0007] (4) Poor processing effect of long documents: Directly vectorizing the entire long document will result in coarse semantic granularity and reduced recall accuracy; simple fixed-length segmentation is easy to cut off the context semantics, causing the loss of key information.

[0008] Therefore, there is an urgent need for a knowledge base construction and retrieval method that can dynamically adapt to multiple embedding models, support Chinese word segmentation fields, take into account both semantic recall and keyword recall, and can be maintained at the file granularity. Summary of the Invention

[0009] In view of this, embodiments of the present invention provide a method for constructing and hybridly retrieving a Chinese vector knowledge base compatible with multiple embedding models, to solve the technical problems in the prior art such as dimension mismatch, interface incompatibility, insufficient accurate Chinese retrieval capability, and complex knowledge base maintenance when switching embedding models. The method includes: Receive configuration information for the embedded model runtime, and construct the corresponding embedding function based on the model type in the configuration information; Dynamically obtain the vector dimension of the current embedded model output, and construct a vector database table corresponding to the vector dimension based on the vector dimension; The imported document is recursively segmented to generate multiple text fragments. Each text fragment is then subjected to Chinese word search and segmentation to generate segmented text. Adjacent text fragments retain overlapping segments. The current embedding model is invoked to generate document vectors for each text segment. The segmented text, document vector, file identifier, segment number, and each text segment corresponding to each text segment are written into the vector database table to generate a record corresponding to each text segment and write it into the vector database. A full-text search index is created for the segmented text to construct a Chinese vector knowledge base. The system receives a query request from a user, generates a query vector and segmented query text for the query request, calls the embedding function corresponding to the current embedding model, performs a mixed retrieval in the Chinese vector knowledge base based on the query vector and the segmented query text, and returns the query results.

[0010] This invention also provides a Chinese vector knowledge base construction and hybrid retrieval system compatible with multiple embedding models, to solve the technical problems in the prior art such as dimension mismatch when switching embedding models, interface incompatibility, insufficient accurate Chinese retrieval capabilities, and complex knowledge base maintenance. The system includes: The embedded model configuration module is used to receive configuration information for the embedded model at runtime and construct the corresponding embedding function according to the model type in the configuration information. The database table construction module is used to dynamically obtain the vector dimension of the current embedded model output and construct a vector database table corresponding to the vector dimension based on the vector dimension. The text segmentation module is used to recursively segment the imported document to generate multiple text fragments. Each text fragment is then used for Chinese word search and segmentation to generate segmented text. Adjacent text fragments retain overlapping segments. The database construction module is used to call the current embedding model to generate document vectors for each text segment, write the segmented text, document vector, file identifier, segment number and each text segment corresponding to each text segment into the vector database table to generate a record corresponding to each text segment and write it into the vector database, and create a full-text search index for the segmented text to build a Chinese vector knowledge base; The retrieval and result return module is used to receive query requests input by users, generate query vectors and query segmented text for the query request, call the embedding function corresponding to the current embedding model, perform a mixed retrieval in the Chinese vector knowledge base based on the query vectors and the query segmented text, and return the query results.

[0011] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned method for constructing and retrieving a Chinese vector knowledge base compatible with any of the multiple embedding models, thereby solving the technical problems in the prior art such as dimension mismatch, interface incompatibility, insufficient Chinese accurate retrieval capability, and complex knowledge base maintenance when switching embedding models.

[0012] This invention also provides a computer-readable storage medium storing a computer program that executes the above-described method for constructing and retrieving a Chinese vector knowledge base compatible with any of the multiple embedding models, in order to solve the technical problems in the prior art such as dimension mismatch, interface incompatibility, insufficient Chinese accurate retrieval capability, and complex knowledge base maintenance when switching embedding models.

[0013] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least: 1. This application constructs vector database tables by configuring normalization and dynamically building them at runtime, while also supporting OpenAI-compatible cloud interfaces and Ollam local interfaces; 2. This application uses dynamic vector dimension detection to automatically adapt to the output dimension of different models. When switching models, there is no need to modify the code or rebuild the knowledge base, which greatly improves the flexibility of the system. 3. This application adopts a recursive segmentation strategy based on multi-level delimiter priority, combined with the overlapping mechanism of adjacent segments, to reduce semantic breaks caused by hard segmentation and improve the accuracy of long document retrieval. 4. This application improves the accuracy of retrieval of Chinese keywords, proper nouns, and numbers by using Chinese word segmentation fields and full-text indexing; Five applications utilize a combination of vector retrieval and full-text retrieval to improve the overall recall effect of semantically relevant and keyword-related content. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a method for constructing and retrieving a Chinese vector knowledge base compatible with multiple embedding models, provided by an embodiment of the present invention. Figure 2 This is a flowchart of a vector dimension dynamic detection method provided in an embodiment of the present invention; Figure 3 This is a flowchart of a document entry process provided in an embodiment of the present invention; Figure 4 This is a flowchart of a hybrid retrieval process provided in an embodiment of the present invention. Detailed Implementation

[0016] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0017] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In this embodiment of the invention, a method for constructing and hybridly retrieving a Chinese vector knowledge base compatible with multiple embedding models is provided, such as... Figure 1 As shown, the method includes: Receive configuration information for the embedded model runtime, and construct the corresponding embedding function based on the model type in the configuration information; Dynamically obtain the vector dimension of the current embedded model's output, and construct a vector database table corresponding to the vector dimension based on the vector dimension; The imported document is recursively segmented to generate multiple text fragments. Each text fragment is then subjected to Chinese word search and segmentation to generate segmented text. Adjacent text fragments retain overlapping segments. The current embedding model is invoked to generate document vectors for each text segment. The segmented text, document vector, file identifier, segment number, and each text segment corresponding to each text segment are written into the vector database table to generate a record corresponding to each text segment and write it into the vector database. A full-text search index is created for the segmented text to construct a Chinese vector knowledge base. The system receives a query request from a user, generates a query vector and segmented query text for the query request, calls the embedding function corresponding to the current embedding model, performs a mixed retrieval in the Chinese vector knowledge base based on the query vector and the segmented query text, and returns the query results.

[0019] In practice, the system receives the configuration for the embedded model runtime. This configuration includes at least the embedded model type, model name, service address, and key. Embedded model types include OpenAI-compatible types and Ollam types. The configuration fields are normalized, supporting multiple field aliases. For example, the model name can come from `model`, `modelName`, or `name`; the service address can come from `baseUrl`, `base_url`, or `url`; and the key can come from `apiKey`, `api_key`, or `key`.

[0020] In practical implementation, in order to simultaneously adapt to the cloud-based OpenAI compatible interface and the local Ollama service, the following steps are proposed to achieve embedded model configuration parsing and runtime construction: Receive configuration information for the embedded model runtime, and construct the corresponding embedding function based on the model type in the configuration information.

[0021] Specifically, when the configuration type is Ollam, the system constructs an Ollam embedding function and converts the service address into an Ollam-recognizable host address; when the configuration type is an OpenAI-compatible interface, the system constructs an OpenAI-compatible embedding function and sets the model name, service address, and key; at the same time, the embedding runtime is warmed up to detect interface configuration errors in advance.

[0022] In practical implementation, to automatically adapt to the output dimensions of different models, the following steps are proposed to achieve dynamic detection of vector dimensions and construction of the knowledge base table structure: Dynamically obtain the vector dimension of the current embedded model's output, and construct a vector database table corresponding to the vector dimension based on the vector dimension.

[0023] Specifically, before creating the knowledge base table, the system submits a probe text to the current embedded model to obtain a probe vector; the system calculates the length of the probe vector and uses the vector length as the vector dimension of the current knowledge base table; if the probe fails, the system records the abnormal path and generates a prompt message based on the error content.

[0024] The system creates a vector database table based on the vector dimension of dynamic detection. Each record in the table includes at least: the original text fragment "content", the Chinese segmented text "content_segmented", the vector field "vector", the file identifier "file_id", the chunk number "chunk_idx", and the creation time "create_date".

[0025] The dimension of the vector field is dynamically determined when the vector dimension is obtained.

[0026] For example, such as Figure 2 As shown, the system receives a knowledge base creation request, reads the embedding model configuration information, and analyzes whether the configuration type has changed. If it has changed, the system clears the runtime vector dimension cache and the opened knowledge base tables, and reconstructs the embedding function. If the configuration type has not changed, the original embedding function is used to submit the probe text. If a valid vector is returned after the probe text is submitted, the length of the valid vector is calculated and cached as the vector dimension of the current knowledge base table, and the knowledge base table is created. If no valid vector is returned, the system records the error message generated by the abnormal link.

[0027] In practice, to reduce semantic breaks caused by hard segmentation and improve the accuracy of long document retrieval, the following steps are proposed to recursively segment the imported document to generate multiple text fragments. Each text fragment is then subjected to Chinese search word segmentation to generate segmented text, wherein adjacent text fragments retain overlapping segments: For imported documents whose length exceeds the preset slice size, a multi-level delimiter priority strategy is used for segmentation. The overlapping segments between adjacent text fragments after segmentation are retained to a preset number of overlapping characters. The multi-level delimiter priorities are in the following order from high to low: paragraph and line break, Chinese punctuation and English punctuation, space and character length.

[0028] Specifically, when importing a document, the system normalizes the document text. If the document length exceeds the preset slice size, it is recursively sliced ​​according to delimiters such as paragraphs, line breaks, Chinese punctuation, English punctuation, and spaces. The system uses a multi-level delimiter priority during slicing. Slicing is prioritized by paragraphs and line breaks, followed by Chinese and English punctuation, and finally by spaces or character length. Overlap is preserved between adjacent slices.

[0029] For any adjacent chunk_i and chunk_i+1, the overlapping region satisfies: tail(chunk_i, overlap) ~= head(chunk_i+1, overlap) This strategy allows entities, definitions, conditional statements, and conclusion statements that cross paragraph boundaries to appear simultaneously in adjacent segments, reducing semantic breaks caused by hard segmentation.

[0030] A preset number of overlapping characters are maintained between adjacent text segments after segmentation to reduce recall loss caused by semantic slicing. For each text segment, the system performs Chinese search word segmentation to obtain several segmentation tokens, which are then connected with spaces to form content_segmented (i.e., segmented text).

[0031] In practice, the system calls the current embedding model to generate document vectors for all text fragments in batches. The system then writes the text fragments, segmented text, document vectors, file identifiers, and segment numbers into the vector database. After writing, the system creates a full-text search index on the `content_segmented` field. The index uses space-separated tokens as search units to adapt to the Chinese word-segmented text.

[0032] For example, with Figure 3 For example, the system performs document segmentation and index creation. After reading the document text, the system normalizes the text. If the document text length exceeds the preset slice length, it recursively segments the text according to paragraphs, line breaks, Chinese punctuation, English punctuation, and spaces. A preset number of overlapping characters is retained between adjacent text segments after segmentation. If the document text length does not exceed the preset slice length, the document text is treated as a single text segment. The normalized text segments are then subjected to Chinese search word segmentation to generate `content_segmented`. The system calls the corresponding embedding model to generate document vectors from the text segments and writes the text segments, segmented text, document vectors, file identifiers, and segment numbers into the vector database. A full-text search index is created or refreshed on the `content_segmented` field generated by the Chinese search word segmentation, and the number of successfully added segments is returned.

[0033] In practice, to improve the overall recall effect of semantically relevant content and keyword-related content, the following steps are taken to achieve a mixed retrieval based on the query vector and the query segmented text in the Chinese vector knowledge base: The first retrieval result is obtained based on the similarity between the query vector and the text fragment; Based on the query segmented text, a full-text search is performed on the segmented text to obtain a second search result; The query results are derived based on the first and second search results, wherein the query results include at least a text fragment, a file identifier, a fragment number, and a creation time.

[0034] Specifically, when the system receives a user query, it first performs embedding vector generation on the query text to obtain the query vector; at the same time, it performs Chinese word segmentation on the query text to obtain the segmented query text.

[0035] The system performs hybrid retrieval in the vector database, which includes: 1. Similarity retrieval based on query vectors; 2. Full-text search based on query-segmented text; 3. Perform a comprehensive sorting of the two search signals.

[0036] If the user specifies a set of file identifiers, the system adds a file_id filter condition during the search, returning results only within the specified file range. The returned results include at least the text fragment, file identifier, fragment number, and creation time.

[0037] For example, such as Figure 4 As shown, after receiving a user's query request, the system normalizes the query text, performs embedding vector generation and Chinese word segmentation on the query text, obtaining the corresponding query vector and segmented query text. Based on the query vector and segmented query text, vector similarity retrieval and full-text retrieval are performed respectively. When performing vector similarity retrieval, if the user specifies a file identifier, a file_id filter condition is added, returning search results within the specified file range. The two types of search results are then mixed and sorted, and the number of returned results is limited to obtain the final mixed search result.

[0038] In practice, to achieve document-level deletion, filtering, and reconstruction, and reduce the maintenance cost of the knowledge base, management is also carried out through file identifiers: Based on the file identifier in the operation instructions, corresponding operations are performed on the text fragments corresponding to the file identifier in the Chinese vector knowledge base, thereby achieving the maintenance of the Chinese vector knowledge base at the file granularity.

[0039] Specifically, this operation can include deletion, updating, and other similar actions. For example, the system supports deleting text fragments in the knowledge base by file_id. During deletion, the system constructs a file_id filtering expression, deletes the corresponding fragment record, and recreates or refreshes the full-text search index. Through the file_id mechanism, the system can process only the data corresponding to a specified file when documents are updated, deleted, or re-imported.

[0040] In practical implementation, to improve the compatibility of semantic recall and precise recall in Chinese scenarios, a dual-channel semantic-lexical representation structure is used: The same text fragment is simultaneously written into both the semantic channel and the lexical channel. The semantic channel stores the semantic vector corresponding to the text fragment, which is used to achieve semantic matching of synonyms, hyponyms and hypernyms, and contextual relevance. The semantic matching result provides data for obtaining the first search result. The lexical channel stores a word segmentation inverted index built based on the text fragment. The word segmentation inverted index is used to achieve matching of nouns, equipment numbers, product models, legal provisions, and fixed phrases. The matching result provides data for obtaining the second search result.

[0041] Specifically, the system constructs a dual-channel representation for each document fragment: R_i=<content_i, segment_i, vector_i, file_id, chunk_idx, create_date> Where content_i represents the original text channel, segment_i represents the lexical channel obtained by Chinese search word segmentation, and vector_i represents the semantic channel generated by the embedding model.

[0042] The semantic channel is used to capture synonyms, hyponyms, and contextual relevance to achieve semantic matching, providing data for similarity retrieval between query vectors and text fragments. The lexical channel is used to capture precise hits of proper nouns, numbers, product models, legal clauses, and phrases, providing data for retrieval of query segmented text within segmented text generated by Chinese search word segmentation. By simultaneously writing the same fragment into both the semantic and lexical channels, the problems of simple vector retrieval being insensitive to precise keywords and simple full-text retrieval being insensitive to semantic rewriting are avoided.

[0043] In practice, when the configuration signature of the embedded model carried by the new query request is inconsistent with the configuration signature of the currently running embedded model, the relevant running data of the currently running embedded model is cleared, and the corresponding embedded model is used for retrieval based on the configuration signature of the embedded model carried by the new query request.

[0044] Specifically, the system will construct an embedded configuration signature by combining the embedded model type, model name, service address, and key configuration digest: Sig_embed=hash(type||model||baseUrl||keyStatus) When a new request carries a Sig_embed that does not match the current runtime cache signature, the system immediately clears the embedding function, vector dimension cache, database connection object, and open table object. This mechanism prevents runtime state contamination between different embedding models or different server endpoints.

[0045] In practical implementation, the Chinese vector knowledge base construction and hybrid retrieval method further abstracts the hybrid retrieval results into the following scoring model: Score(q,d)=alpha Sim(V_q,V_d)+beta Lex(q_seg,d_seg)+gamma F(file_id) Where V_q is the query vector, V_d is the document segmentation vector, Sim represents the vector similarity, q_seg represents the query segmented text, d_seg represents the document segmentation field, Lex represents the lexical matching score, F(file_id) represents the file filtering or file range constraint, and alpha, beta, and gamma are weight parameters.

[0046] In practical implementation, the vector database can have built-in hybrid retrieval and ranking capabilities, and the system can also perform secondary re-ranking after returning the results. The above scoring model is used to describe the synergistic relationship between semantic recall, lexical recall, and document constraints.

[0047] In practice, the specific implementation process of the above-mentioned Chinese vector knowledge base construction and hybrid retrieval method is as follows: Step S1: The system runs on the Python sidecar service of the desktop application, uses LanceDB as the vector database, and employs a search-mode word segmentation tool for Chinese word segmentation.

[0048] Step S2: Configure the embedded model on the settings page. The configuration can be of OpenAI compatible type, including model name, base_url, and api_key; or of Ollam type, including model name and host. The system will pass the configuration to the knowledge base service.

[0049] After receiving the configuration, the knowledge base service first parses and normalizes the configuration fields. If the configuration changes, it clears the cached embedded runtime, vector dimensions, and open tables to prevent old configurations from polluting the new configuration.

[0050] Step S3: Create a knowledge base. The system calls the embedding model to vectorize the probe text. If the returned vector length is 1024, the system creates a table with a vector dimension of 1024; if the returned vector length is 1536, the system creates a table with a vector dimension of 1536. Thus, the knowledge base table structure automatically adapts to the embedding model.

[0051] Step S4: Import the file. The system reads the parsed text content of the file. If the text length does not exceed the preset slice size, it is treated as a single segment; if it exceeds the preset size, a recursive text splitter is used to split the text by paragraphs, line breaks, punctuation, and spaces, while retaining the preset number of overlapping characters.

[0052] For each text segment, the system calls the Chinese word segmentation module to generate the `content_segmented` field and then calls the embedding model to generate a vector. The system writes `content`, `content_segmented`, `vector`, `file_id`, `chunk_idx`, and `create_date` as a single record into the vector database.

[0053] Step S5: After the document is written, the system creates a full-text search index on the `content_segmented` field. Since `content_segmented` already contains space-separated Chinese tokens, the full-text search engine can perform searches by token.

[0054] Step S6: Knowledge Base Query and File Deletion. When querying the knowledge base, the system performs embedding and Chinese word segmentation on the query text simultaneously. Then, a hybrid search is performed in the vector database: semantic matching is performed using the query vectors, and full-text matching is performed using the segmented text. If the user selects to retrieve only certain files, the system appends a file_id filtering expression. Finally, the system returns the text fragments with the highest scores. When deleting a file, the system deletes all related fragments based on the file_id corresponding to that file and refreshes the full-text index. Thus, file-level updates can be completed without rebuilding the entire knowledge base.

[0055] In this embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for constructing and retrieving a Chinese vector knowledge base compatible with any of the multiple embedding models.

[0056] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0057] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that executes the above-described method for constructing and retrieving Chinese vector knowledge bases compatible with any of the multiple embedding models.

[0058] Based on the same inventive concept, this invention also provides a multi-embedded model compatible Chinese vector knowledge base construction and hybrid retrieval system, as described in the following embodiments. Since the principle of solving the problem based on the multi-embedded model compatible Chinese vector knowledge base construction and hybrid retrieval system is similar to that of the multi-embedded model compatible Chinese vector knowledge base construction and hybrid retrieval method, the implementation of the multi-embedded model compatible Chinese vector knowledge base construction and hybrid retrieval system can refer to the implementation of the multi-embedded model compatible Chinese vector knowledge base construction and hybrid retrieval method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0059] In specific implementation, the Chinese vector knowledge base construction and hybrid retrieval system based on multi-embedding model compatibility includes: The embedded model configuration module is used to receive configuration information for the embedded model at runtime and construct the corresponding embedding function according to the model type in the configuration information. The database table construction module is used to dynamically obtain the vector dimension of the current embedded model's output and construct a vector database table corresponding to the vector dimension based on the vector dimension. The text segmentation module is used to recursively segment the imported document to generate multiple text fragments. Each text fragment is then used for Chinese word search and segmentation to generate segmented text. Adjacent text fragments retain overlapping segments. The database construction module is used to call the current embedding model to generate document vectors for each text segment, write the segmented text, document vector, file identifier, segment number and each text segment corresponding to each text segment into the vector database table to generate a record corresponding to each text segment and write it into the vector database, and create a full-text search index for the segmented text to build a Chinese vector knowledge base; The retrieval and result return module is used to receive query requests input by users, generate query vectors and query segmented text for the query request, call the embedding function corresponding to the current embedding model, perform a mixed retrieval in the Chinese vector knowledge base based on the query vectors and the query segmented text, and return the query results.

[0060] In one embodiment, a database table construction module is used to submit probe vectors, which are vectorized from probe text output, to the current embedded model, and to calculate the length of the probe vectors as the vector dimension.

[0061] In one embodiment, the text segmentation module is used to segment imported document text that exceeds a preset slice size using a multi-level delimiter priority strategy, and the overlapping segments retained between adjacent text segments after segmentation are a preset number of overlapping characters. The multi-level delimiter priorities are in the following order from high to low: paragraph and line break, Chinese punctuation and English punctuation, space and character length.

[0062] In one embodiment, the retrieval and result return module is used to obtain a first retrieval result based on the similarity between the query vector and the text fragment; to perform a full-text search on the text fragment based on the query segmented text to obtain a second retrieval result; and to derive the query result based on the first retrieval result and the second retrieval result, wherein the query result includes at least the text fragment, file identifier, fragment number, and creation time.

[0063] In one embodiment, the priority order of the inspectors corresponding to different violation types of Handlers, from high to low, is as follows: inspectors corresponding to social exemptions, technical revocations, and evidence quality. The social exemptions include special vehicles and traffic police commands; the technical revocations include driving under a green light. The higher the priority of the inspector, the fewer fields it reads from the structured semantic evidence when judging violations.

[0064] In one embodiment, the system further includes: The granularity maintenance module is used to perform corresponding operations on the text fragments corresponding to the file identifier in the Chinese vector knowledge base according to the file identifier in the operation instruction, so as to realize the maintenance of the Chinese vector knowledge base at the file granularity.

[0065] The dual-channel representation module is used to simultaneously write the same text fragment into a semantic channel and a lexical channel. The semantic channel is used to capture synonyms, hyponyms, and contextual relevance, while the lexical channel is used to capture proper nouns, numbers, product models, regulatory clauses, and phrases.

[0066] The failure control module is used to clear the relevant running data of the currently running embedded model when the configuration signature of the embedded model carried by the new query request is inconsistent with the configuration signature of the currently running embedded model, and to perform retrieval using the corresponding embedded model based on the configuration signature of the embedded model carried by the new query request.

[0067] The embodiments of the present invention achieve the following technical effects: It supports both OpenAI's cloud-compatible interface and Ollam's local interface, automatically adapting to the output dimensions of different models. Switching models requires no code modification or knowledge base reconstruction, significantly improving system flexibility. It can recognize synonyms and context-related semantic content, as well as accurately match precise keywords such as proper noun numbers, greatly improving recall performance. It enables document-level deletion, updating, and importing, reducing knowledge base maintenance costs and improving maintenance efficiency. It adopts a recursive segmentation strategy with multi-level delimiter priority, combined with an overlapping mechanism of adjacent segments, reducing semantic breaks caused by hard segmentation and improving the accuracy of long document retrieval.

[0068] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0069] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing and retrieval a Chinese vector knowledge base compatible with multiple embedding models, characterized in that, include: Receive configuration information for the embedded model runtime, and construct the corresponding embedding function based on the model type in the configuration information; Dynamically obtain the vector dimension of the current embedded model output, and construct a vector database table corresponding to the vector dimension based on the vector dimension; The imported document is recursively segmented to generate multiple text fragments. Each text fragment is then subjected to Chinese word search and segmentation to generate segmented text. Adjacent text fragments retain overlapping segments. The current embedding model is invoked to generate document vectors for each text segment. The segmented text, document vector, file identifier, segment number, and each text segment corresponding to each text segment are written into the vector database table to generate a record corresponding to each text segment and write it into the vector database. A full-text search index is created for the segmented text to construct a Chinese vector knowledge base. The system receives a query request from a user, generates a query vector and segmented query text for the query request, calls the embedding function corresponding to the current embedding model, performs a mixed retrieval in the Chinese vector knowledge base based on the query vector and the segmented query text, and returns the query results.

2. The method for constructing and hybrid retrieving Chinese vector knowledge bases as described in claim 1, characterized in that, Dynamically obtain the vector dimensions of the current embedded model's output, including: Submit probe text to the current embedded model to output a vectorized probe vector, and calculate the length of the probe vector as the vector dimension.

3. The method for constructing and hybrid retrieving Chinese vector knowledge bases as described in claim 1, characterized in that, The imported document is recursively segmented to generate multiple text fragments, including: For imported documents whose length exceeds the preset slice size, a multi-level delimiter priority strategy is used for segmentation. The overlapping segments between adjacent text fragments after segmentation are retained to a preset number of overlapping characters. The multi-level delimiter priorities are in the following order from high to low: paragraph and line break, Chinese punctuation and English punctuation, space and character length.

4. The method for constructing and hybrid retrieving Chinese vector knowledge bases as described in claim 1, characterized in that, Based on the query vector and the segmented query text, a hybrid retrieval is performed in the Chinese vector knowledge base, including: The first retrieval result is obtained based on the similarity between the query vector and the text fragment; Based on the query segmented text, a full-text search is performed on the segmented text to obtain a second search result; The query results are derived based on the first and second search results, wherein the query results include at least a text fragment, a file identifier, a fragment number, and a creation time.

5. The Chinese vector knowledge base construction and hybrid retrieval method as described in claim 1 further includes: Based on the file identifier in the operation instructions, corresponding operations are performed on the text fragments corresponding to the file identifier in the Chinese vector knowledge base, thereby achieving the maintenance of the Chinese vector knowledge base at the file granularity.

6. The Chinese vector knowledge base construction and hybrid retrieval method as described in claim 1 further includes: The same text fragment is simultaneously written into both the semantic channel and the lexical channel. The semantic channel stores the semantic vector corresponding to the text fragment, which is used to achieve semantic matching of synonyms, hyponyms and hypernyms, and contextual relevance. The semantic matching result provides data for obtaining the first search result. The lexical channel stores a word segmentation inverted index built based on the text fragment. The word segmentation inverted index is used to achieve matching of nouns, equipment numbers, product models, legal provisions, and fixed phrases. The matching result provides data for obtaining the second search result.

7. The Chinese vector knowledge base construction and hybrid retrieval method as described in claim 1 further includes: When the configuration signature of the embedded model carried in the new query request is inconsistent with the configuration signature of the currently running embedded model, the relevant running data of the currently running embedded model is cleared, and the corresponding embedded model is used for retrieval based on the configuration signature of the embedded model carried in the new query request.

8. A Chinese vector knowledge base construction and hybrid retrieval system compatible with multiple embedding models, characterized in that, include: The embedded model configuration module is used to receive configuration information for the embedded model runtime and construct the corresponding embedding function according to the model type in the configuration information. The database table construction module is used to dynamically obtain the vector dimension of the current embedded model output and construct a vector database table corresponding to the vector dimension based on the vector dimension. The text segmentation module is used to recursively segment the imported document to generate multiple text fragments. Each text fragment is then used for Chinese word search and segmentation to generate segmented text. Adjacent text fragments retain overlapping segments. The database construction module is used to call the current embedding model to generate document vectors for each text segment, write the segmented text, document vector, file identifier, segment number and each text segment corresponding to each text segment into the vector database table to generate a record corresponding to each text segment and write it into the vector database, and create a full-text search index for the segmented text to build a Chinese vector knowledge base; The retrieval and result return module is used to receive query requests input by users, generate query vectors and query segmented text for the query request, call the embedding function corresponding to the current embedding model, perform a mixed retrieval in the Chinese vector knowledge base based on the query vectors and the query segmented text, and return the query results.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the Chinese vector knowledge base construction and hybrid retrieval method compatible with multiple embedding models as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the Chinese vector knowledge base construction and hybrid retrieval method compatible with any one of claims 1 to 7.

Images