Index updating method and device of vector database, medium and equipment

Abstract

The invention discloses an index updating method and device for a vector database, a medium and equipment, and belongs to the field of index updating. Original text data and a first vector set in the vector database are firstly obtained, and an original text is vectorized and standardized through a preset semantic coding model to obtain a second vector set; after a third vector is screened out through semantic consistency verification, the third vector is embedded into the initial index map by adopting an improved incremental HNSW strategy to obtain a first vector index map, and then edge weight pruning and adjacent edge reconstruction optimization are performed through semantic similarity to obtain a second vector index map; and finally, calculating the quantitative proportion of the third vector to the first vector, and if the quantitative proportion reaches a preset threshold value, extracting nodes meeting semantic neighborhood conditions by taking the third vector as a center to form a local sub-graph and reconstructing the local sub-graph to obtain an updated vector database index. The problem that the index of the database cannot be accurately and efficiently updated in the prior art is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of index updating, and more particularly to an index updating method, apparatus, medium, and device for a vector database. Background Technology

[0002] With the rapid evolution of artificial intelligence technology, large-scale pre-trained language models, represented by BERT and GPT, have been widely applied in the field of natural language processing, driving vectorized text representation methods to become the core technical support for knowledge expression, information retrieval, and intelligent question answering systems in professional domains. By transforming text data into high-dimensional semantic vectors, semantic similarity can be accurately calculated in vector space, thereby achieving more intelligent retrieval and matching. Vector databases have thus emerged, becoming a key infrastructure for carrying high-dimensional semantic vectors and supporting Approximate Nearest Neighbor (ANN) queries.

[0003] Current mainstream vector database indexing technologies largely rely on offline training with batch data. Typical solutions include Facebook's FAISS, the Hierarchical Navigation Small World (HNSW) algorithm based on multi-layer graph structures, and Annoy. Among them, HNSW, with its characteristics of "multi-layer sparse graph navigation + low-level precise nearest neighbor connection," balances query accuracy and efficiency, and has become the preferred indexing solution for large-scale vector scenarios. It can quickly locate target vectors in high-dimensional space through skip-search and has good scalability.

[0004] However, existing vector indexing technologies still face significant technical bottlenecks in the dynamic updating and semantic stability maintenance of professional domain knowledge bases: First, graph-structured indexes, represented by HNSW, only establish connecting edges through local nearest neighbor selection during incremental updates, without global structural optimization. This easily leads to the formation of "weak edges" with weak semantic relevance, causing the index graph structure to degrade with increasing incremental cycles. This is especially true in high-dimensional and sparse professional domain vector spaces, significantly reducing retrieval accuracy. Second, document updates in professional domains (such as power secondary systems and medical clinical guidelines) are often accompanied by terminology revisions or semantic changes, and existing technologies lack semantic consistency. First, the lack of validation mechanisms means that inserting new vectors directly without validation can easily introduce redundant or conflicting data, leading to misjudgments or semantic drift in search results. Second, traditional index updates often use a full reconstruction mode, requiring a complete index rebuild for each new data addition, which is time-consuming and resource-intensive, failing to meet the business needs of real-time or near real-time updates for professional domain knowledge bases. Third, existing technologies also suffer from problems such as the lack of dynamic edge weight management (edge ​​relationships cannot reflect changes in semantic density) and coarse-grained index reconstruction (only full or single-point adjustments, resulting in low resource utilization). These problems prevent existing technologies from accurately and efficiently updating database indexes. Summary of the Invention

[0005] This invention provides a method, apparatus, medium, and device for updating the index of a vector database, thereby solving the problem that existing technologies cannot accurately and efficiently update the index of a database.

[0006] Firstly, this application provides an index update method for a vector database, including:

[0007] Obtain the original text data and the first vector set in the vector database;

[0008] Based on a preset semantic encoding model, the original text data is semantically vectorized, and the generated initial vectors are standardized to obtain a second vector set.

[0009] The second vector set and the first vector set are semantically consistent, and each third vector that meets the preset requirements is selected.

[0010] Based on the improved incremental HNSW insertion strategy, each of the third vectors is embedded into a preset initial vector index graph to obtain a first vector index graph;

[0011] Based on the semantic similarity method, the first vector index graph is optimized by edge weight pruning and adjacent edge reconstruction to obtain the second vector index graph;

[0012] Calculate the ratio of the total number of each third vector to the total number of vectors in the first vector set;

[0013] If the ratio reaches a preset reconstruction ratio threshold, nodes in the second vector index graph that satisfy the preset neighborhood condition with the semantic similarity to the third vector are extracted as centers to form a local subgraph. Semantic nearest neighbor connections are performed on the local subgraph to complete the reconstruction and obtain the updated vector database index.

[0014] This application vectorizes and standardizes the original text using a pre-defined semantic encoding model. On one hand, it leverages the model's ability to capture domain-specific semantics to ensure accurate vector semantic expression. On the other hand, standardization eliminates the interference of vector magnitude differences on similarity calculations, laying a high-quality data foundation for subsequent semantic consistency verification and index construction, and avoiding retrieval errors caused by insufficient vector quality. Secondly, it performs semantic consistency verification between the second vector set and the first vector set. By selecting third vectors that meet semantic requirements, redundant, conflicting, or semantically drifting vectors are directly eliminated, avoiding the problem of "unverified vector insertion leading to index semantic pollution" in existing technologies from the source. This significantly improves the overall semantic accuracy and stability of the index and reduces the risk of retrieval misjudgment. Furthermore, based on improvements... When embedding the third vector using the incremental HNSW insertion strategy, it avoids the defect of "too many weak edges due to local point selection" in traditional HNSW incremental insertion by refining the nearest neighbor search and target level allocation layer by layer. This makes the topological structure of the first vector index graph more reasonable and ensures the efficiency of the retrieval path. Subsequently, through semantic similarity-driven edge weight pruning and adjacent edge reconstruction, low-quality weak edges are deleted and high-semantic-confidence adjacency relationships are reconstructed, further optimizing the connectivity and robustness of the index graph. Finally, by calculating the proportion of the third vector, a local subgraph reconstruction is triggered, rather than using a full reconstruction mode. This accurately optimizes local areas where the index structure density changes, while avoiding the high resource overhead of full reconstruction, significantly reducing index maintenance costs and meeting the needs of real-time or near real-time updates for professional domain knowledge bases. This application effectively solves the problem that existing technologies cannot accurately and efficiently update database indexes.

[0015] Furthermore, based on a preset semantic encoding model, the original text data is semantically vectorized, and the generated initial vectors are standardized to obtain a second vector set, specifically:

[0016] The title, body text, and appendix content in the original text data are split, and the body text is divided into several standardized text segments according to the preset paragraph length.

[0017] The standardized text segments are input into a preset semantic encoding model so that the semantic encoding model performs semantic encoding on each standardized text segment and generates corresponding initial vectors; wherein, the semantic encoding model is a Sentence-BERT, MPNet or MiniLM model fine-tuned by a preset domain corpus.

[0018] Perform L2 norm normalization on each initial vector and calculate the square root of each dimension component of each initial vector to obtain the second vector set.

[0019] This application first splits and segments the original text data to obtain standardized text segments. This preprocessing step ensures the standardization of the text data, providing a unified input format for subsequent semantic encoding. Next, these standardized text segments are input into a semantic encoding model (such as Sentence-BERT, MPNet, or MiniLM) fine-tuned with a domain corpus to generate initial vectors. Fine-tuning the domain corpus enables the model to more accurately capture specialized terminology and domain-specific semantic information, thereby generating more representative and accurate initial vectors. Finally, the initial vectors are L2 norm normalized to ensure that all vectors have a uniform length, i.e., mapped to a unit hypersphere. This helps eliminate the impact of vector magnitude differences on similarity calculation, improving the consistency and accuracy of similarity calculation. Through this series of processes, the resulting second vector set more accurately reflects the semantic information of the text, providing a high-quality vector foundation for subsequent index construction and retrieval, thereby improving the retrieval accuracy and stability of the entire system.

[0020] Furthermore, the step of performing semantic consistency verification between the second vector set and the first vector set, and selecting each third vector whose semantics meet the preset requirements, specifically involves:

[0021] Select a vector to be verified from the second set of vectors;

[0022] Calculate the semantic similarity between the vector to be verified and each vector in the first vector set; wherein the semantic similarity is obtained by vector dot product operation;

[0023] The first semantic similarity value with the highest semantic similarity is compared with a preset semantic threshold. If the first semantic similarity value is less than the semantic threshold, it is determined that the semantics of the vector to be verified meet the preset requirements, and the vector to be verified is used as the third vector.

[0024] Until all vectors in the second vector set have been verified, all vectors whose semantics meet the preset requirements are used as the third vectors.

[0025] This application selects a vector to be verified from the second vector set and calculates its semantic similarity to each vector in the first vector set. Semantic similarity is obtained through vector dot product, a simple and effective similarity calculation method that reflects the closeness of two vectors in the semantic space. Next, the first semantic similarity value with the highest value is selected and compared with a preset semantic threshold. The preset semantic threshold is a key parameter used to determine whether the semantic similarity between the vector to be verified and existing vectors is within an acceptable range. If the first semantic similarity value is less than the preset semantic threshold, it indicates that the semantic similarity between the vector to be verified and existing vectors is low, meaning the vector is semantically new and non-repetitive, meeting the preset requirements, and is therefore classified as a third vector. Finally, the above steps are repeated until all vectors in the second vector set have been verified. Ultimately, all vectors that semantically meet the preset requirements are retained as third vectors. This process effectively filters out semantically repetitive or highly similar vectors, preventing these vectors from interfering with the index graph structure, thereby improving the semantic accuracy and stability of the entire vector database and providing a higher-quality data foundation for subsequent index construction and retrieval.

[0026] Furthermore, the improved incremental HNSW insertion strategy embeds each third vector into a preset initial vector index graph to obtain a first vector index graph, specifically as follows:

[0027] An exponential distribution sampling function is constructed based on preset normalization coefficients, preset attenuation coefficients, and the maximum allowed number of layers in the preset initial vector index graph.

[0028] The probability of each third vector being assigned to each level in the initial vector index graph is calculated using the exponential distribution sampling function, and the target insertion level of each third vector is determined based on the probability.

[0029] Starting from the highest level of the initial vector index graph, and using the third vector as a reference, a predetermined number of candidate nodes with the highest semantic similarity are searched among the existing nodes in the highest level to obtain the highest level candidate node set; based on the highest level candidate node set, the search continues to the next level to search for candidate nodes that are better than the third vector, and this process is repeated layer by layer until the bottom level of the initial vector index graph is reached to obtain the bottom level candidate node set.

[0030] From the bottom-level candidate node set, the top N candidate nodes in terms of semantic similarity are selected to form a nearest neighbor node set; where N is a positive integer.

[0031] Establish bidirectional connection edges between the third vector and each node in the set of nearest neighbors, add the third vector to the corresponding target insertion level of the initial vector index graph, until all third vectors are embedded in the initial vector index graph, and obtain the first vector index graph.

[0032] This application constructs an exponential distribution sampling function based on preset normalization coefficients, preset decay coefficients, and a preset maximum allowed number of layers in the initial vector index graph. This sampling function calculates the probability of each third vector being assigned to each layer in the initial vector index graph and determines its target insertion layer. This layered allocation method can reasonably distribute new vectors to different layers, laying the foundation for efficient subsequent retrieval. Next, starting from the highest layer of the initial vector index graph, using the third vector as a benchmark, a preset number of candidate nodes with the highest semantic similarity are searched among the existing nodes in the highest layer, resulting in a set of candidate nodes for the highest layer. Then, based on the set of candidate nodes for the highest layer, the search continues down to the next layer, continuing to search for candidate nodes that are better than the third vector, repeating layer by layer until the bottom layer of the initial vector index graph is reached, resulting in a set of candidate nodes for the bottom layer. This process leverages the hierarchical structure advantage of HNSW, refining the search range layer by layer, enabling rapid location of the node most similar to the target vector in large-scale data, greatly improving retrieval efficiency. Finally, the top N candidate nodes with the highest semantic similarity from the bottom-level candidate node set are selected to form a nearest neighbor node set. A bidirectional connection is established between the third vector and each node in the nearest neighbor node set, and the third vector is added to the corresponding target insertion level of the initial vector index graph. In this way, the new vector can be reasonably embedded into the index graph while maintaining the structural characteristics and retrieval performance of the index graph. This process continues until all third vectors are embedded in the initial vector index graph, resulting in the first vector index graph. The entire process not only improves the efficiency of vector index construction but also ensures the high performance of the index graph in subsequent retrievals.

[0033] Furthermore, the semantic similarity-based method is used to optimize the first vector index graph by edge weight pruning and adjacent edge reconstruction to obtain the second vector index graph, specifically as follows:

[0034] Traverse all connecting edges in the first vector index graph, calculate the semantic similarity between the vectors corresponding to the two ends of each connecting edge, and use the semantic similarity as the edge weight of the corresponding connecting edge to obtain the first vector index graph with edge weight label.

[0035] Mark the connecting edges whose edge weights are less than the preset weak edge judgment threshold as weak edges, delete all weak edges from the first vector index graph with edge weight markings, and obtain the first vector index graph after deleting weak edges.

[0036] A node in the first vector index graph after deleting weak edges is selected as the current processing node. From the candidate nearest neighbor set of the current processing node, nodes with semantic similarity not lower than the weak edge judgment threshold are selected. The first M nodes are selected after sorting them by semantic similarity from high to low to obtain the target nearest neighbor set of the current processing node; where M is a positive integer; the candidate nearest neighbor set consists of the nodes visited by the current processing node during the third vector embedding process.

[0037] Establish the adjacency edges between the current processing node and each node in the target nearest neighbor node set, and obtain the local graph structure after reconstructing the adjacency edges of a single node;

[0038] The process continues until the adjacent edges of all nodes in the first vector index graph after deleting weak edges are reconstructed. Then, the local graph structure after reconstructing the adjacent edges of all single nodes is summarized to obtain the second vector index graph.

[0039] This application first traverses all connecting edges in the first vector index graph, calculates the semantic similarity between the vectors corresponding to the nodes at both ends of each connecting edge, and uses the semantic similarity as the edge weight of the corresponding connecting edge, resulting in a first vector index graph with edge weight labels. This process provides a quantitative basis for subsequent edge weight evaluation, allowing the semantic relevance of each edge to be clearly represented. Next, connecting edges with edge weights less than a preset weak edge judgment threshold are marked as weak edges, and all weak edges are deleted from the first vector index graph with edge weight labels, resulting in a first vector index graph with weak edges removed. Through this pruning operation, connecting edges with low semantic relevance are removed, reducing noise and redundant information in the index graph, thereby improving the overall semantic quality of the index graph. Then, a node in the first vector index graph with weak edges removed is selected as the current processing node. From the candidate nearest neighbor set of the current processing node, nodes with semantic similarity not lower than the weak edge judgment threshold are selected, and the top M nodes are selected after sorting them by semantic similarity from high to low, resulting in the target nearest neighbor set of the current processing node. This process ensures that the adjacent nodes of each node have high semantic relevance, further optimizing the structure of the index graph. Finally, adjacency edges are established between the current processing node and each node in the target nearest neighbor set, resulting in the local graph structure after single-node adjacency edge reconstruction. This process is repeated until the adjacency edge reconstruction of all nodes in the first vector index graph after deleting weak edges is complete. Finally, the local graph structures after single-node adjacency edge reconstruction are summarized to obtain the second vector index graph. Through this adjacency edge reconstruction mechanism, the semantic consistency and retrieval performance of the index graph are significantly improved, while maintaining the connectivity and coverage of the index graph.

[0040] Further, the calculation of the ratio of the total number of each third vector to the total number of vectors in the first vector set; if the ratio reaches a preset reconstruction ratio threshold, taking each third vector as the center, extracting nodes in the second vector index graph whose semantic similarity to the third vector satisfies the preset neighborhood condition to form a local subgraph, performing semantic nearest neighbor connections on the local subgraph to complete the reconstruction, and obtaining the updated vector database index, specifically:

[0041] Calculate the ratio of the total number of each third vector to the total number of vectors in the first vector set as the update ratio;

[0042] The update ratio is compared with the preset reconstruction ratio threshold. If the update ratio is greater than or equal to the preset reconstruction ratio threshold, nodes are selected in the second vector index graph with each third vector as the center according to the preset neighborhood conditions. The selected nodes are combined with the third vector to obtain the local subgraph corresponding to each third vector.

[0043] Select a local subgraph as the current processing subgraph, delete the original connecting edges of all nodes in the current processing subgraph, and obtain the local subgraph after deleting the original edges;

[0044] For each node in the local subgraph after deleting the original edges, recalculate the semantic similarity between each node and other nodes in the subgraph, sort the nodes by semantic similarity from high to low, and select a preset number of nodes as the target nearest neighbor set.

[0045] Establish bidirectional connection edges between each node in the current processing subgraph and the nodes in the corresponding target nearest neighbor set to obtain the subgraph structure after the single local subgraph reconstruction is completed;

[0046] Continue until all local subgraphs are reconstructed, resulting in all reconstructed local subgraphs;

[0047] The updated vector database index is obtained by summarizing all the reconstructed local subgraphs and the remaining parts of the second vector index graph that were not included in the local subgraphs.

[0048] This application first calculates the ratio of the total number of each third vector to the total number of vectors in the first vector set as the update ratio. This ratio reflects the degree of influence of the newly inserted vector on the entire index structure. The update ratio is compared with a preset reconstruction ratio threshold. If the update ratio is greater than or equal to the preset reconstruction ratio threshold, it indicates that the newly inserted vector has a significant impact on the index structure, and local reconstruction is required to optimize the index structure.

[0049] Next, using each third vector as a center, nodes are selected in the second vector index graph according to preset neighborhood conditions. The selected nodes are then combined with the third vectors to obtain the local subgraph corresponding to each third vector. This process determines the local region that needs to be reconstructed, ensuring the targeted and efficient nature of the reconstruction operation.

[0050] Next, a local subgraph is selected as the current processing subgraph, and all original connection edges of nodes within the current processing subgraph are deleted, resulting in a local subgraph with the original edges removed. This step provides a clean starting point for subsequent reconnections, avoiding the influence of old connections on the new structure.

[0051] For each node in the local subgraph after deleting the original edges, the semantic similarity between each node and other nodes in the subgraph is recalculated. A predetermined number of nodes are then selected as the target nearest neighbor set, sorted by semantic similarity from highest to lowest. This process, based on the latest semantic similarity information, redetermines the optimal adjacency relationship for each node, thereby optimizing the structure of the local subgraph.

[0052] Establish bidirectional connections between each node in the current processing subgraph and nodes in the corresponding target nearest neighbor set, resulting in the subgraph structure after single local subgraph reconstruction. This step completes the reconstruction of the local subgraph, ensuring that the connections between each node have higher semantic relevance.

[0053] Repeat the above steps until all local subgraphs are reconstructed, resulting in all reconstructed local subgraphs. Finally, summarize all reconstructed local subgraphs and the remaining parts of the second vector index graph that were not included in the local subgraphs to obtain the updated vector database index. This process ensures the connectivity and consistency of the entire index structure, while optimizing index performance through local reconstruction.

[0054] Secondly, this application provides an index updating device for a vector database. The index updating device for the vector database includes:

[0055] The acquisition module is used to acquire the original text data and the first vector set in the vector database.

[0056] The processing module is used to perform semantic vectorization processing on the original text data based on a preset semantic encoding model, and to standardize the generated initial vectors to obtain a second vector set.

[0057] The verification module is used to perform semantic consistency verification between the second vector set and the first vector set, and to filter out each third vector whose semantics meet the preset requirements;

[0058] An embedding module is used to embed each of the third vectors into a preset initial vector index graph based on an improved incremental HNSW insertion strategy, thereby obtaining a first vector index graph.

[0059] An optimization module is used to optimize the first vector index graph by performing edge weight pruning and adjacent edge reconstruction based on the semantic similarity method, so as to obtain the second vector index graph.

[0060] The reconstruction module is used to calculate the ratio of the total number of each third vector to the total number of vectors in the first vector set;

[0061] If the ratio reaches a preset reconstruction ratio threshold, nodes in the second vector index graph that satisfy the preset neighborhood condition with the semantic similarity to the third vector are extracted as centers to form a local subgraph. Semantic nearest neighbor connections are performed on the local subgraph to complete the reconstruction and obtain the updated vector database index.

[0062] The acquisition module of this application acquires the original text data and a first vector set from the vector database, providing the foundational data for subsequent processing. The processing module uses a pre-defined semantic encoding model to perform semantic vectorization on the original text data and standardizes the generated initial vectors to obtain a second vector set. This process ensures high-quality vector generation, providing a solid foundation for subsequent index construction and retrieval.

[0063] Next, the verification module performs semantic consistency verification between the second vector set and the first vector set, filtering out the third vectors whose semantics meet the preset requirements. This verification process effectively filters out vectors with semantic duplication or conflict, ensuring the semantic accuracy and stability of the index.

[0064] Then, the embedding module, based on an improved incremental HNSW insertion strategy, embeds each third vector into a preset initial vector index graph to obtain a first vector index graph. This insertion strategy not only improves the efficiency of vector embedding but also maintains the structural characteristics and retrieval performance of the index graph.

[0065] The optimization module uses a semantic similarity method to perform edge weight pruning and adjacency edge reconstruction on the first vector index graph to obtain the second vector index graph. This optimization process further improves the semantic consistency and retrieval performance of the index graph by removing low-quality connecting edges and reconstructing adjacency relationships.

[0066] Finally, the reconstruction module calculates the ratio of the total number of third vectors to the total number of vectors in the first vector set. If this ratio reaches a preset reconstruction ratio threshold, local subgraphs are formed by extracting nodes from the second vector index graph that satisfy preset neighborhood conditions with respect to the semantic similarity of the third vectors, centered on each third vector. Semantic nearest neighbor connections are then performed on these local subgraphs to complete the reconstruction, resulting in an updated vector database index. This local reconstruction mechanism optimizes the index structure and reduces maintenance costs while maintaining index connectivity and retrieval efficiency, all while avoiding a full index rebuild.

[0067] Thirdly, this application provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform an index update method for a vector database as described above. Its beneficial effects are the same as those of the index update method for a vector database provided in the first aspect of this application.

[0068] Fourthly, this application provides a terminal device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement an index update method for any of the vector databases described in the first aspect. Attached Figure Description

[0069] Figure 1 : A schematic flowchart of an embodiment of the vector database index update method provided in this application;

[0070] Figure 2 : A schematic diagram of an embodiment of the index update process for the vector database provided in this application;

[0071] Figure 3 : A schematic diagram of an embodiment of the index update device for the vector database provided in this application. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] Example 1

[0074] Please refer to Figure 1In order to solve the problem that existing technologies cannot accurately and efficiently update database indexes, this invention provides an index update method for a vector database, including steps S01-S07.

[0075] S01: Obtain the original text data and the first vector set in the vector database.

[0076] In a preferred embodiment of this invention, the step of obtaining the original text data and the first vector set in the vector database specifically involves:

[0077] In this embodiment, the process of obtaining the original text data and the first vector set in the vector database needs to be executed collaboratively from three dimensions: data source location, data filtering and verification, and data extraction and storage, in order to ensure the quality of the underlying data for subsequent semantic vectorization and index updates.

[0078] The acquisition of raw text data needs to be combined with the professional domain requirements of the target application scenario, and collected from conventional data carriers in that domain. For example, in the scenario of a power secondary system knowledge base, text data including equipment operation procedures, maintenance standards, accident analysis reports, and safety specifications can be collected from the power company's document management system, industry supervision platform, or equipment operation and maintenance database. In the scenario of a medical clinical guideline knowledge base, text data such as clinical diagnosis and treatment guidelines, drug instructions, and case analysis documents from medical institutions can be collected. During the collection process, the format of the raw text data needs to be initially identified and classified simultaneously, covering common document formats such as PDF, Word, and TXT. The pre-set file parsing module is used to perform layout parsing on text of different formats, splitting out the structured content such as titles, body text, appendices, and chart descriptions in the text. At the same time, invalid data with damaged format, incomplete content (such as missing key chapters or garbled characters) or non-target domain (such as general text unrelated to power or medical fields) is removed. Only text data with compliant format, complete content, and that meets the professional domain requirements is retained as the input source for subsequent semantic vectorization processing.

[0079] The first vector set in the vector database is essentially a collection of historical semantic vectors already stored in the database. The corresponding original texts are mostly professional documents that have already undergone semantic encoding and been incorporated into the knowledge base. To obtain this set, the storage access interface of the vector database is used to call its historical vector storage module to extract all standardized historical semantic vectors. These vectors include both vector data generated after semantic encoding of past professional documents and valid vectors that have passed semantic consistency verification and been successfully embedded in the index during previous incremental updates. During the extraction process, the obtained historical vectors must undergo data integrity verification to confirm the completeness of the vector dimensions and identification information (such as the document ID and paragraph number of the corresponding original text). If there are issues such as abnormal vector dimensions or missing identifiers, the database's data repair mechanism must be used to supplement or remove abnormal vectors. The resulting complete and valid historical vector set is the first vector set for subsequent semantic consistency verification with newly added vectors.

[0080] The original text data and the first vector set obtained through the above method can be used as raw materials for subsequent semantic vectorization processing and as a benchmark for semantic consistency verification, respectively. This provides high-quality initial data support for the entire vector database index update process, ensuring the accuracy of semantic vectors and the reliability of verification results in subsequent steps.

[0081] S02: Based on a preset semantic encoding model, the original text data is semantically vectorized, and the generated initial vectors are standardized to obtain a second vector set.

[0082] In a preferred embodiment of this invention, the original text data is semantically vectorized based on a preset semantic encoding model, and the generated initial vectors are standardized to obtain a second vector set, specifically as follows:

[0083] In this embodiment, the semantic vectorization and standardization of the original text data based on the preset semantic encoding model is the core step in transforming professional domain text into high-dimensional semantic vectors that can be used for index construction. The specific process revolves around three key steps: text structuring, professional semantic encoding, and vector space standardization, to ensure that the generated second vector set has accurate semantic expression capabilities and a unified calculation benchmark.

[0084] First, it is necessary to clarify the source of the original text data and the preprocessing logic: the original text data can come from various channels, such as the power industry's operating procedures, equipment maintenance standards, accident analysis reports, or internal company operation manuals, R&D documents or meeting minutes, or normative documents issued by industry associations and regulatory agencies. Before entering the system, this data needs to undergo format parsing and paragraph segmentation through a pre-defined file parsing module. This involves first separating the original text data (which has been pre-screened and is formatted correctly and has complete content) into different structural units such as titles, body text, appendices, and chart descriptions, focusing on extracting the body text content that carries core professional semantics. Then, based on the text's logical paragraph division rules (such as punctuation marks and chapter separators), the body text is segmented into standardized text segments of appropriate length. For example, for power regulations, the body text can be segmented into segments corresponding to individual clauses based on "clause numbers," while for medical guidelines, it can be segmented according to logical modules such as "treatment steps" and "medication recommendations." This ultimately forms a standardized text set that can be used for subsequent semantic vectorization processing, ensuring that each text segment contains only a single, complete professional semantic unit, avoiding the impact of semantic mixing on subsequent coding results.

[0085] Subsequently, the pre-set semantic encoding model is activated to perform semantic vectorization processing. This semantic encoding model needs to be fine-tuned and optimized for the characteristics of the corpus in the target professional field. For example, in the power field, professional documents on power secondary systems can be used, and in the medical field, clinical guidelines and pharmacopoeia corpora can be used. Supervised or unsupervised training is performed on basic models such as Sentence-BERT, MPNet, or MiniLM to enable them to accurately capture the semantic associations of professional terms in the field (such as the specific semantics of terms like "relay protection device" and "secondary circuit" in the power field, and "targeted therapy" and "pharmacokinetics" in the medical field). Each standardized text segment obtained from the above preprocessing is input into the fine-tuned semantic encoding model one by one. The model extracts and maps the semantic features of the text segments through its internal Transformer encoder, and finally generates a high-dimensional initial vector for each text segment. The dimension of the initial vector is consistent with the output dimension of the model, and each dimension component of the vector corresponds to the quantitative expression of a specific semantic feature of the text segment.

[0086] More specifically, let the set of text paragraphs to be processed be:

[0087] T = {t1, t2, ..., t} n}

[0088] Among them, t i This represents the i-th text segment to be processed, where n is the total number of text segments.

[0089] Using semantic encoding function f θ For each text segment ti Encode and generate an initial vector:

[0090]

[0091] Among them, f θ This represents the encoding function based on a pre-trained language model and fine-tuned for a specific domain, where θ is the set of model parameters, and v i The semantic vector generated for the i-th text segment has dimension d.

[0092] Finally, all generated initial vectors are standardized to eliminate the interference of vector magnitude differences on subsequent semantic similarity calculations. Specifically, the sum of squares of each dimension component of each initial vector is first calculated, and then the arithmetic square root of this sum is taken to obtain the L2 norm of the initial vector. Subsequently, each dimension component of the initial vector is divided by its corresponding L2 norm to unify the magnitude of the standardized vectors to 1, ensuring that all vectors are mapped to the unit hypersphere. At this point, the geometric direction of the vectors can directly represent the semantic information of the text segment, and the result of the dot product operation between vectors can be directly equated to semantic similarity, providing a unified and accurate calculation benchmark for subsequent consistency verification with the first vector set. After all initial vectors have been standardized, all standardized vectors are associated and stored according to the identification information of the original text segment (such as document ID, paragraph number). The resulting standardized vector set is the second vector set, which will serve as the source of new vectors for the next step of semantic consistency verification.

[0093] More specifically, since the similarity calculation between vectors depends on their orientation information, to ensure the consistency and comparability of cosine similarity calculations, the generated vectors need to be normalized using the L2 norm. The normalized vectors... The calculation formula is as follows:

[0094]

[0095] in, This represents the semantic representation after unit vectorization, ||v i ||2 represents the original vector v i The L2 norm, Represents vector v i The component in the j-th dimension, where d is the total number of dimensions of the vector.

[0096] After this standardization process, all vectors will be mapped onto a unit hypersphere, and their geometric directions will represent the semantic information of the text. This process helps improve the consistency of subsequent similarity calculations and avoids semantic offset problems caused by differences in vector magnitudes.

[0097] After this step is completed, we obtain the normalized second set of vectors:

[0098]

[0099] This set will serve as input data for subsequent semantic consistency verification and graph structure construction.

[0100] After all initial vectors have been normalized to the L2 norm, all standardized vectors are then... The original text segments are associated and stored according to their identification information (such as document ID and paragraph number), and the resulting standardized vector set is the second vector set.

[0101] The input for this step is the original text data and the standardized text segments (i.e., set T) processed by the file parsing module; the output is a second set of vectors after domain-specific semantic encoding and L2 norm normalization, and the text identifier information corresponding to each vector (used for text tracing during subsequent semantic consistency verification). This output will directly serve as the input data source for step S03 (semantic consistency verification between the second and first vector sets), providing a high-quality vector foundation for subsequent index construction.

[0102] S03: Perform semantic consistency verification between the second vector set and the first vector set, and filter out each third vector whose semantics meet the preset requirements.

[0103] In a preferred embodiment of this invention, the step of performing semantic consistency verification between the second vector set and the first vector set, and selecting each third vector whose semantics meet preset requirements, specifically involves:

[0104] In this embodiment, the second vector set and the first vector set are semantically consistent and the third vector is selected. The core is to eliminate the parts of the newly added vector that are redundant, conflicting or semantically drifting with the historical vectors by quantifying the semantic similarity comparison, so as to ensure that the vectors finally embedded in the index have professional domain semantic validity. The specific operation needs to be carried out around the logical closed loop of "vector-by-vector verification - similarity calculation - threshold determination - result selection".

[0105] First, the fundamental conditions for verification are clarified: each vector in the second vector set has been normalized to the L2 norm, and the historical vectors in the first vector set are also standardized semantic vectors. Both exist in a unified unit hyperspherical vector space. Therefore, the dot product operation between vectors can be directly equated to semantic similarity, providing a computational basis for the accuracy of the verification results. The verification process requires processing each vector in the second vector set as the object to be verified, that is, sequentially selecting a single vector from the second vector set as the current vector to be verified, and then calculating the semantic similarity between this vector and all historical vectors in the first vector set.

[0106] In the similarity calculation stage, for the current vector to be verified, each historical vector in the first vector set is traversed, and the semantic similarity value between the two is obtained through vector dot product operation. Since the vectors are all standardized, the value of the dot product result is in the range of [-1, 1]. The larger the value, the more similar the texts corresponding to the two vectors are semantically. After each round of similarity calculation with all historical vectors in the first vector set, the maximum value needs to be extracted from all the obtained similarity values. This maximum value can intuitively reflect the degree of association between the current vector to be verified and the semantically most similar content in the existing knowledge base. If the maximum value is too high, it means that the text content corresponding to the vector to be verified has a high degree of semantic overlap with the historical documents, which may be redundant data; if the maximum value is too low, it means that the vector to be verified is semantically new and valid data.

[0107] The process then proceeds to the threshold determination stage. A semantic compliance threshold is preset (this threshold needs to be set in conjunction with the semantic redundancy tolerance of the professional domain; for example, in the scenario of power regulations, it is necessary to avoid misjudgments caused by subtle differences in the wording of clauses, and in the scenario of medical guidelines, it is necessary to strictly exclude duplicate treatment recommendations). The maximum similarity value corresponding to the current vector to be verified is compared with this threshold. If the maximum similarity value is less than the preset threshold, it is determined that the semantics of the current vector to be verified do not significantly overlap with the historical vectors in the first vector set, there is no semantic conflict or redundancy problem, and it meets the semantic requirements of the professional domain knowledge base, and it is marked as a valid vector. If the maximum similarity value is greater than or equal to the preset threshold, it is determined that the text content corresponding to the current vector to be verified is highly similar to or completely duplicates the semantics of the historical document, which is redundant data or has the risk of semantic expression conflict, and it needs to be removed from the vectors to be processed. If there are special scenarios where there is partial semantic correlation but it needs to be retained (such as partial revisions in document version updates), the vector merging logic can also be triggered to merge the vector to be verified with the corresponding similar historical vectors.

[0108] More specifically, let the normalized second vector set be:

[0109]

[0110] The first vector set is:

[0111]

[0112] For each second vector Calculate its cosine similarity with all first vectors in the database, defined as follows:

[0113]

[0114] in, Represents the second vector With the first vector Cosine similarity between them Representing vectors The component in the l-th dimension, Representing vectors The components are on the same dimension, where d is the dimension of the vector.

[0115] In this step, for each second vector Calculate its relationship with all The similarity is taken as the maximum value:

[0116]

[0117] If the maximum similarity Greater than the preset threshold τ dup If the vectors are semantically duplicated or highly similar, they are considered semantically duplicated or not inserted into the index; otherwise, they are considered semantically valid new vectors and retained for subsequent insertion processes. The judgment condition can be expressed as follows:

[0118] like Then retain Otherwise, discard or merge.

[0119] Where, τ dup The semantic similarity deduplication threshold is set to (0,1), and it is recommended to set it to 0.95 in this application to identify redundant content or paragraphs with overlapping versions. This value can be configured according to the actual business semantic redundancy tolerance.

[0120] Through the semantic consistency verification mechanism described above, the system can dynamically remove new data that highly overlaps with the vector semantics in the historical database, preventing excessive duplication from causing excessive local density in the graph structure, thereby ensuring the balance of the overall topology of the graph index and improving the accuracy and efficiency of subsequent searches.

[0121] Repeat the steps of "selecting the vector to be verified—calculating the similarity with all historical vectors—extracting the maximum similarity—threshold comparison and judgment" until all vectors in the second vector set have been verified. Finally, summarize all vectors marked as valid to form a vector set that semantically conforms to the requirements of the professional domain. This set will be the third vectors used for subsequent index embedding. Through this verification process, redundant and conflicting vectors can be blocked from entering the index structure at the source, avoiding problems such as retrieval misjudgment and path degradation in the subsequent index graph due to semantic pollution, while ensuring the semantic rigor and consistency of the professional domain knowledge base.

[0122] S04: Based on the improved incremental HNSW insertion strategy, each of the third vectors is embedded into a preset initial vector index graph to obtain a first vector index graph.

[0123] In a preferred embodiment of this invention, the improved incremental HNSW insertion strategy embeds each third vector into a preset initial vector index graph to obtain a first vector index graph, specifically as follows:

[0124] In this embodiment, each third vector is embedded into a preset initial vector index graph based on an improved incremental HNSW (Hierarchical Navigation Small World) insertion strategy. The core is to achieve efficient embedding of the third vector in the existing graph structure through hierarchical precise allocation, layer-by-layer nearest neighbor refinement search, and bidirectional edge structure update, while avoiding the "weak edge" problem that is easy to produce by traditional incremental insertion. The specific process needs to be executed in an orderly manner around four steps: "hierarchical allocation - intra-layer search - nearest neighbor selection - edge structure update".

[0125] First, the preset initial vector index graph is a pre-constructed HNSW multi-layer graph structure. This structure contains several levels, with sparse nodes in higher levels for fast navigation and dense nodes in lower levels for accurate nearest neighbor matching. The graph already stores the nodes corresponding to the first vector set and the bidirectional connecting edges between nodes. For each third vector to be embedded, its target insertion level in the initial vector index graph must be determined first. Level allocation is achieved through a constructed exponential distribution sampling function. This function combines the current maximum allowed number of levels in the initial vector index graph, a decay coefficient to control the number of nodes in higher levels, and a normalization coefficient to ensure that the probability sum is 1. The function calculates the probability of the third vector being assigned to each level, and finally determines the target insertion level of the vector based on the probability distribution results. This ensures that most third vectors are concentrated in lower levels to guarantee retrieval accuracy, while a few vectors are assigned to higher levels to maintain navigation efficiency.

[0126] After completing the hierarchical allocation, the intra-layer nearest neighbor search process is initiated: taking the highest layer of the initial vector index graph as the search starting point, and using the currently embedded third vector as the benchmark, the highest-level candidate nodes are selected from the existing nodes through semantic similarity calculation (consistent with the calculation logic after vector standardization in the previous stage, implemented based on vector dot product) to form a set of candidate nodes for the highest layer; then, based on this set of candidate nodes for the highest layer, the search proceeds to the next level, where semantic similarity is searched only for the adjacent nodes of the highest-level candidate nodes, further refining the candidate node set for that level; this logic is followed layer by layer downwards, with each level's candidate nodes being selected based on the candidate nodes of the previous level, until the bottom layer of the initial vector index graph is reached, finally obtaining the bottom-level candidate node set with the best semantic similarity, ensuring the accuracy of the nearest neighbor search results.

[0127] At the bottom layer of the initial vector index graph, the candidate node set is sorted from high to low semantic similarity. A predetermined number of nodes are selected to form the nearest neighbor set of the current third vector. The nodes in this set are most semantically related to the third vector and can be used to construct stable connection edges. Then, bidirectional connection edges are established between the third vector and each node in the nearest neighbor set. That is, these nodes are added to the adjacent edge set of the third vector, and the current third vector is also added to the adjacent edge set of each nearest neighbor node, realizing symmetrical update of the edge structure. If the number of adjacent edges of a certain nearest neighbor node exceeds the predetermined limit (to maintain the sparsity of the graph structure), the edge update requirement of that node is temporarily stored and will be uniformly optimized in subsequent edge weight pruning steps.

[0128] More specifically, the preset initial vector index graph is a pre-constructed HNSW multi-layer graph structure, which contains several levels. The high-level nodes are sparse and used for fast navigation, while the low-level nodes are dense and used for accurate nearest neighbor matching. The graph already stores the nodes corresponding to the first vector set and the bidirectional connection edges between the nodes.

[0129] For each third vector v to be embedded i (i.e., a qualified vector after semantic consistency verification) needs to have its target insertion level l in the initial vector index graph determined by a hierarchical sampling strategy (i.e., the highest level at which the vector should be inserted). Level l is generated by sampling using an exponential distribution function, as shown in the following formula:

[0130] P(l) = p0·e -λl ,l∈{0,1,...,L max}

[0131] Where P(l) represents the probability that the vector is assigned to the l-th layer, p0 is the normalization coefficient used to make the sum of all probabilities equal to 1, λ is the decay coefficient used to control the rate of decrease in the number of high-level nodes, and L max This indicates the maximum number of layers allowed in the current graph structure.

[0132] The core effect of this sampling strategy is that most third vectors are concentrated at lower levels (e.g., l=0 or l=1) to form dense connections and ensure retrieval accuracy; only a small number of vectors are assigned to higher levels (e.g., l is close to L). max This is used to maintain the fast navigation capability of the graph structure, thereby balancing the search performance of the index with the space storage efficiency.

[0133] After completing the hierarchical allocation, the intra-layer nearest neighbor search process is initiated: starting from the highest level of the initial vector index graph, the search begins by finding the third vector v to be embedded. i As a benchmark, filter nodes from the existing nodes at the highest level that are related to v. i The set of candidate nodes with the highest semantic similarity is formed by selecting a predetermined number of candidate nodes.

[0134] In this step, semantic similarity is still calculated using cosine similarity (since all vectors have been normalized to L2 norm, cosine similarity is equivalent to the vector dot product), as shown in the following formula:

[0135]

[0136] in, Indicates the addition of a new node With existing nodes The similarity is such that all vectors are L2 normalized unit vectors.

[0137] At the bottom layer of the initial vector index graph, the candidate node set is sorted from high to low semantic similarity, and a preset number M nodes (M is a constant for the number of connecting edges, which can be configured according to the index precision requirements, such as 16 or 32) are selected to form the current third vector v. i The set of nearest neighbors N i The definition is as follows:

[0138]

[0139] in, Represents a node The set of nearest neighbors selected in this layer, M is the number of connecting edges, which is usually a constant, such as 16 or 32, and can be flexibly configured according to the index precision requirements.

[0140] Then establish v i With N i The bidirectional connection edges between each node, i.e.:

[0141]

[0142] Here, E represents the set of edges in the current graph structure.

[0143] If the number of connected edges of a certain neighboring node exceeds the preset limit, the edge update requirements of that node are temporarily stored and will be uniformly optimized in the subsequent "edge weight pruning mechanism" in step S05 to avoid the graph structure becoming bloated due to the overload of the number of edges.

[0144] Repeating the above steps of "hierarchical allocation—intra-layer search—nearest neighbor selection—edge structure update," all third vectors are sequentially embedded into the initial vector index graph. During this process, the node set and edge set of the initial vector index graph are updated in real time after each third vector is embedded. After all third vectors are embedded, a graph structure containing the original historical nodes, newly added third vector nodes, and the updated edge structure is obtained. This graph structure is the first vector index graph. Through this improved incremental insertion strategy, the embedding process of third vectors not only relies on the multi-layer navigation advantages of HNSW to ensure efficiency, but also reduces the generation of low-quality connections through layer-by-layer refined search and bidirectional edge construction, laying a stable graph structure foundation for subsequent index optimization.

[0145] S05: Based on the semantic similarity method, the first vector index graph is optimized by edge weight pruning and adjacent edge reconstruction to obtain the second vector index graph.

[0146] In a preferred embodiment of this invention, the second vector index graph is obtained by performing edge weight pruning and adjacent edge reconstruction optimization on the first vector index graph based on the semantic similarity method, specifically as follows:

[0147] In this embodiment, the first vector index graph is optimized by edge weight pruning and adjacent edge reconstruction based on the semantic similarity method. The core is to solve the problems of loose edge structure and unstable retrieval path that may exist in the first vector index graph after incremental embedding by quantifying the semantic quality of connecting edges, eliminating low-quality "weak edges" and reconstructing high-confidence adjacency relationships. Finally, a second vector index graph with better topology is formed. The specific process needs to be promoted step by step according to the logic of "edge weight calculation - weak edge pruning - adjacent edge reconstruction - global optimization summary".

[0148] First, edge weight calculation is performed: The first vector index graph already contains numerous connecting edges between nodes. These edges are mostly constructed based on local nearest neighbor relationships during third vector embedding and have not yet undergone semantic quality verification. Therefore, all connecting edges in the first vector index graph need to be traversed. For each connecting edge, the standardized semantic vectors corresponding to its two endpoints are extracted (whether original historical node vectors or newly added third vectors, all have undergone L2 norm normalization). The semantic similarity between the two vectors is calculated through vector dot product. Since the vectors reside in a unified unit hypersphere space, the dot product result is directly equivalent to the semantic similarity. This result is used as the "edge weight" of the corresponding connecting edge to quantify the semantic association strength of each connecting edge. The larger the edge weight value, the more semantically similar the nodes at both ends of the connecting edge are, and the higher the connection quality; conversely, a smaller weight indicates a low-quality connection.

[0149] After calculating edge weights, the weak edge pruning stage begins. A weak edge judgment threshold is preset, which needs to be set in conjunction with the semantic association requirements of the specific domain. This threshold is used to distinguish whether a connecting edge has a minimum semantic confidence. All connecting edges with edge weights are traversed in the first vector index graph. The edge weight of each edge is compared with the weak edge judgment threshold. If the edge weight of a connecting edge is less than the threshold, it is judged as a "weak edge." The semantic association between the two nodes of such edges is weak, which not only increases the storage burden of the index graph but may also lead to incorrect paths during retrieval, resulting in decreased retrieval accuracy. Therefore, all weak edges need to be directly deleted from the first vector index graph. After pruning, the number of adjacent edges of some nodes may be significantly reduced, or even missing. High-quality adjacency relationships need to be added through subsequent reconstruction steps to maintain the connectivity of the index graph.

[0150] Then, the adjacency edge reconstruction optimization is initiated: For each node in the pruned first vector index graph, its exclusive candidate nearest neighbor set is extracted. This set is not randomly selected, but rather derived from nodes visited by the node during the previous third vector embedding process (including candidate nodes searched during embedding, nodes with established temporary connections, etc.), ensuring that the nodes in the set have a previous semantic association with the current processing node, reducing invalid computations during reconstruction. From this candidate nearest neighbor set, nodes with a semantic similarity to the current processing node not lower than the weak edge judgment threshold (i.e., meeting the minimum semantic confidence requirement) are selected. These nodes are sorted from high to low semantic similarity, and the first preset number of nodes are selected to form the target nearest neighbor set of the current processing node. Afterward, new adjacency edges are established between the current processing node and each node in the target nearest neighbor set, and the edge weights of the new adjacency edges need to be verified again (to ensure they are not lower than the weak edge judgment threshold), forming the adjacency edge reconstruction result for a single node.

[0151] More specifically, in a graph structure, each edge can be assigned a weight value based on the semantic similarity between the connected nodes, serving as a measure of connection strength. Let the nodes... and nodes There is a connecting edge e between them ij Then the edge weight w ij The definition is as follows:

[0152]

[0153] Among them, w ij Representing edge e ij The edge weights take values ​​in the interval [-1, 1], with larger values ​​indicating more similar semantics; and Let be the L2 normalized vector representations of node i and node j, respectively, and · denotes the vector dot product operation, i.e., cosine similarity.

[0154] To improve the semantic aggregation quality of graphs, this application sets an edge weight threshold τ. weak This is used to distinguish whether there are low-quality connections in the structure. For any node Its set of connecting edges is denoted as E. i ={e ij If there exists an edge e ij ∈E i satisfy:

[0155] w ij <τ weak

[0156] If an edge is deemed "weak" and lacks sufficient semantic support, it should be removed from the graph structure. This pruning operation ensures that the retained adjacency relationships meet the minimum semantic confidence requirement, thereby maintaining the semantic consistency of each node in the graph.

[0157] Where, τ weak To preset the lower limit threshold of edge weights, it is recommended to take a value in the range of [0.5, 0.7], which can be flexibly configured according to the semantic complexity of the professional field.

[0158] After the pruning operation is completed, the node The final adjacency set should satisfy the following conditions:

[0159]

[0160] This constraint ensures that all connecting edges of a node are high-confidence semantic connections, thereby avoiding semantic deviations or invalid jumps in the index path.

[0161] After weak edge pruning, in order to maintain the traversability and nearest neighbor coverage of the graph, this invention introduces a local reconnection mechanism after each pruning, which is used to reselect nodes with higher semantic similarity ranking from the current node candidate set (such as the set of nodes visited in the previous insertion step) to supplement the edge set.

[0162] Specifically, let M′ be the number of neighbor targets after reconstruction, and select from the candidate set C i Select from those that satisfy:

[0163]

[0164] in, Let C represent the reselected adjacency set after pruning. i Let M be the set of candidate nearest neighbor nodes, and M′ be the target value for the number of adjacencies, which is recommended to be consistent with the original number of nearest neighbors M.

[0165] Following the logic described above, perform edge weight pruning and adjacent edge reconstruction operations on all nodes in the first vector index graph. During this process, the adjacent edge set of each node needs to be updated in real time to ensure that the reconstructed edge structure retains only high-quality semantic connections. After all nodes have been optimized, summarize the update results of all nodes to obtain an index graph with a more compact edge structure, stronger semantic relevance, and stable connectivity. This index graph is the second vector index graph. Through this optimization process, low-quality connections in the index graph can be eliminated, reducing retrieval interference. At the same time, high-quality adjacent edges can be added through reconstruction, maintaining the sparsity and robustness of the index graph, and providing a better graph structure foundation for subsequent possible local subgraph reconstruction or direct retrieval.

[0166] S06: Calculate the ratio of the total number of each third vector to the total number of vectors in the first vector set.

[0167] In a preferred embodiment of this invention, the calculation of the ratio of the total number of each third vector to the total number of vectors in the first vector set specifically involves:

[0168] In this embodiment, the ratio of the total number of each third vector to the total number of vectors in the first vector set is calculated. The core purpose is to provide an objective basis for decision-making on whether to trigger the reconstruction of the index local subgraph by quantifying the scale ratio of the newly added effective vectors relative to the historical vectors, and to avoid the problems of "over-reconstruction" or "under-reconstruction" caused by the unknown scale of the newly added vectors. The specific operation needs to be carried out in three steps: "key quantity statistics - ratio calculation - result meaning association".

[0169] First, precise statistics on the number of two core vectors are performed: On the one hand, the third vectors are valid new vectors selected after the initial semantic consistency verification. These vectors have been stored in the temporary vector cache module through identification management (such as assigning a unique vector ID and marking it as "valid"). The total number of each third vector can be obtained by traversing the vector list of this cache module and counting the total number of vectors with the "valid" mark. During the counting process, invalid counts caused by temporary storage anomalies (such as duplicate storage or incorrect marking) must be excluded to ensure the accuracy of the number of new vectors. On the other hand, the first vector set consists of historical semantic vectors that have been stably stored in the vector database. Its total number needs to be obtained by calling the metadata query interface of the vector database. This interface can directly return the total number of all compliant vectors in the historical vector storage module (excluding invalid historical vectors with dimension anomalies or missing marks). If the database supports batch storage, the number of historical vectors in each batch also needs to be summarized to ensure that the total number of the first vector set can truly reflect the vector cardinality of the existing index.

[0170] After completing the statistical analysis of the two key quantities, a ratio calculation is performed: using the total number of the third vector as the numerator and the total number of the first vector set as the denominator, a division operation is performed to obtain the ratio between the two. This ratio represents the scale of the newly added effective vectors relative to the historical vectors. For example, if the total number of the third vector is 1000 and the total number of the first vector set is 5000, then the ratio is 0.2 (i.e., 20%). It is important to note that a certain number of decimal places (e.g., 4 decimal places) must be retained during the calculation to avoid loss of precision due to integer division, ensuring accuracy when comparing with the preset reconstruction ratio threshold later.

[0171] The core significance of this ratio result lies in its direct reflection of the impact of newly added effective vectors on the existing index graph structure. A low ratio indicates that the newly added vectors only account for a small portion of the existing index, having little impact on the topological structure and semantic density of the index graph, thus requiring no large-scale reconstruction. Conversely, a high ratio indicates that the newly added vectors have significantly altered the local semantic distribution of the existing index, potentially leading to an imbalance in the edge density and a decrease in retrieval path efficiency in some regions of the index graph. In this case, subsequent local subgraph reconstruction is necessary to optimize the index structure. Therefore, this ratio calculation step is a crucial decision-making node connecting "initial index optimization (edge ​​pruning and reconstruction)" and "deep index optimization (local subgraph reconstruction)," providing a quantitative basis for the selection of subsequent index maintenance strategies.

[0172] S07: If the ratio reaches the preset reconstruction ratio threshold, take each third vector as the center, extract the nodes in the second vector index graph whose semantic similarity with the third vector meets the preset neighborhood condition to form a local subgraph, perform semantic nearest neighbor connection on the local subgraph to complete the reconstruction, and obtain the updated vector database index.

[0173] In a preferred embodiment of this invention, if the ratio reaches a preset reconstruction ratio threshold, nodes in the second vector index graph whose semantic similarity to the third vector satisfies a preset neighborhood condition are extracted, centered on each third vector, to form a local subgraph. Semantic nearest neighbor connections are then performed on the local subgraph to complete the reconstruction, resulting in an updated vector database index. Specifically:

[0174] In this embodiment, when the ratio of the total number of the third vectors to the total number of the first vector set reaches the preset reconstruction ratio threshold, it indicates that the newly added effective vectors have significantly affected the local semantic distribution and topological structure of the second vector index graph. If the original index structure is maintained, it may lead to a decrease in the retrieval path efficiency or semantic association distortion in the area surrounding the newly added vectors. At this time, it is necessary to achieve accurate optimization through local subgraph reconstruction to avoid the high resource overhead and long time consumption brought about by full reconstruction. The specific process revolves around "local subgraph construction - subgraph internal edge structure reconstruction - global index summary".

[0175] First, local subgraphs are extracted and constructed: each third vector is used as the center of the subgraph (since the third vector is a newly added semantic carrier, its surrounding nodes are most directly affected by changes in semantic distribution). Nodes are selected in the second vector index graph according to preset neighborhood conditions. The preset neighborhood conditions use semantic similarity as the core criterion, that is, nodes with a semantic similarity to the current third vector not lower than a preset neighborhood threshold are selected. These nodes have a close semantic relationship with the third vector and together constitute the "semantic association region" affected by the newly added vector. At the same time, in order to avoid the reconstruction efficiency being reduced due to the excessive size of a single local subgraph, an upper limit is set on the number of nodes in each subgraph. If the number of selected nodes exceeds this upper limit, they are sorted from high to low semantic similarity, and the first preset number of nodes are included in the subgraph. Finally, a dedicated local subgraph centered on a single third vector and containing nodes with high semantic association is formed. If there is node overlap in the local subgraphs corresponding to multiple third vectors (that is, a node belongs to the semantic association region of multiple third vectors at the same time), the association between the node and the subgraph to which the third vector with the highest semantic similarity belongs is retained first, and will be handled uniformly in subsequent reconstruction.

[0176] Next, semantic nearest neighbor reconstruction is performed within the local subgraph: For each local subgraph, all original connecting edges of nodes within the subgraph are first deleted. These original edges may be low-quality edges remaining after previous incremental insertions or edge weight pruning, or edges whose semantic relevance has decreased due to the addition of a third vector. Continuing to retain them will affect the retrieval accuracy within the subgraph. After deleting the original edges, for each node in the subgraph, its semantic similarity with all other nodes in the subgraph is recalculated (the calculation logic is the same as before, based on the dot product operation of standardized vectors). After sorting the nodes by semantic similarity from high to low, a preset number of nodes are selected as the target nearest neighbor set for that node. The number of nodes selected needs to balance the connectivity and sparsity of the subgraph to avoid too many adjacent edges leading to redundant retrieval paths, or too few adjacent edges leading to isolated nodes. Subsequently, bidirectional connecting edges are established between the current node and each node in the target nearest neighbor set, and the edge weights of the new connecting edges need to be verified (to ensure that they are not lower than the preset weak edge judgment threshold) to prevent low-quality edges from being reintroduced into the subgraph, ultimately forming the reconstructed subgraph edge structure.

[0177] More specifically, first, let's clarify the quantification conditions for triggering the reconstruction: Suppose the total number of nodes in the current second vector index graph is N (N = total number of nodes in the first vector set + total number of embedded third vectors, i.e., the sum of all valid nodes since the last index optimization), and the number of new vectors (third vectors) to be added in this reconstruction is N. new The local subgraph reconstruction process is triggered when the proportion of newly added nodes to the total number of existing nodes meets the following conditions:

[0178]

[0179] Where γ is the threshold value for triggering local reconstruction, ranging from (0,1), and is typically set to 0.1 (i.e., triggered when more than 10% of new nodes are added); N new This represents the number of new nodes inserted since the last structural optimization, where N is the total number of nodes in the current graph.

[0180] After triggering the reconstruction, with each third vector v i As the center of the subgraph (v i To add a semantic carrier (whose surrounding nodes are most directly affected by changes in semantic distribution), nodes are selected in the second vector index graph using a dual constraint of "semantic similarity threshold + node number limit" to construct a node set for the local subgraph. (i.e., the local influence region corresponding to the third vector), is defined as follows:

[0181]

[0182] in, Represents nodes Similarity exceeding the threshold τ in semantic space radius The adjacent area, Let τ be the cosine similarity value between the two. radius The lower bound of similarity defined for local neighborhoods.

[0183] To control computational complexity, a maximum node limit k can be set for this neighborhood. max ,Right now:

[0184]

[0185] Where, k max The upper limit for the number of nodes in each local subgraph, typically ranging from 100 to 500, is determined by memory capacity and vector density.

[0186] In determining the subgraph region Then, the system will re-execute the local nearest neighbor connection logic for the nodes within that region to generate a better adjacency structure. This process includes:

[0187] 1. Delete the original edges of all nodes within the region;

[0188] 2. Recalculate the value of each node. The set of nearest neighbor nodes in the array;

[0189] 3. Construct new edges according to the HNSW hierarchical connection method or a simplified symmetric nearest neighbor strategy.

[0190] Specifically, for any node within the region Its new adjacency set It can be represented as:

[0191]

[0192] in, Represents a node The set of neighboring nodes reconnected after local reconstruction, M′ represents the target value of the number of connections after reconstruction, which is usually consistent with or slightly higher than the number of edges M in the insertion phase to enhance structural stability.

[0193] The reconstructed edges still need to undergo edge weight pruning to ensure that the retained edges meet the semantic confidence threshold.

[0194]

[0195] Among them, w jk The semantic similarity weights between new edges are determined by the edge weights. If these weights are less than the weak edge threshold τ, then... weak If the condition is not met, then the edge will not be established.

[0196] Repeat the "subgraph construction - edge structure reconstruction" operation until all local subgraphs corresponding to the third vectors have been reconstructed. Finally, summarize all reconstructed local subgraphs and the remaining parts of the second vector index graph that were not included in the local subgraphs. The parts not included are areas not significantly affected by the new third vectors; their original edge structures still have high semantic confidence and do not require reconstruction. Integrate the reconstructed local subgraphs with the unreconstructed parts by node identifiers, ensuring logically coherent semantic relationships and a complete topological structure between nodes. The resulting index graph is the updated vector database index. This index optimizes key areas affected by the new vectors through local reconstruction while retaining the effective parts of the original structure, achieving a balance between index update efficiency and quality, and can directly support subsequent high-precision vector retrieval needs.

[0197] like Figure 2 As shown, Figure 2 The process of updating the vector database index is demonstrated: After starting from "Start", the original text data is input. First, the text is constructed and standardized in step S1. Then, the semantic consistency verification mechanism in step S2 is used to determine whether the vectors are duplicated or conflicted. If so, they are discarded or merged. Otherwise, the improved incremental HNSW graph insertion strategy in step S3 is used. Then, the edge weight pruning and adjacency optimization mechanism is executed in step S4, and it is determined whether the local reconstruction trigger condition is met. If it is met, the local graph structure reconstruction mechanism in step S5 is used, and the process ends.

[0198] In summary, this embodiment vectorizes and standardizes the original text using a pre-defined semantic encoding model. On one hand, it leverages the model's ability to capture domain-specific semantics to ensure accurate vector semantic expression. On the other hand, standardization eliminates the interference of vector length differences on similarity calculations, laying a high-quality data foundation for subsequent semantic consistency verification and index construction, and avoiding retrieval errors caused by insufficient vector quality. Secondly, semantic consistency verification is performed between the second vector set and the first vector set. By selecting third vectors that meet semantic requirements, redundant, conflicting, or semantically drifting vectors are directly eliminated, avoiding the problem of "unverified vector insertion leading to index semantic pollution" in existing technologies from the source. This significantly improves the overall semantic accuracy and stability of the index and reduces the risk of retrieval misjudgment. Furthermore, based on… The improved incremental HNSW insertion strategy, when embedding the third vector, avoids the drawback of "too many weak edges due to local point selection" in traditional incremental HNSW insertion by refining the nearest neighbor search and target level allocation layer by layer. This makes the topological structure of the first vector index graph more reasonable and ensures the efficiency of the retrieval path. Subsequently, through semantic similarity-driven edge weight pruning and adjacent edge reconstruction, low-quality weak edges are deleted and high-semantic-confidence adjacency relationships are reconstructed, further optimizing the connectivity and robustness of the index graph. Finally, local subgraph reconstruction is triggered by calculating the proportion of the third vector, rather than using a full reconstruction mode. This approach accurately optimizes local areas where the index structure density changes while avoiding the high resource overhead of full reconstruction, significantly reducing index maintenance costs and meeting the needs of real-time or near real-time updates for professional domain knowledge bases. This application effectively solves the problem that existing technologies cannot accurately and efficiently update database indexes.

[0199] Example 2

[0200] Please refer to Figure 3 This is an index update device for a vector database provided in the embodiments of this application.

[0201] In this embodiment, the vector database index update device includes an acquisition module 10, a processing module 20, a verification module 30, an embedding module 40, an optimization module 50, and a reconstruction module 60.

[0202] Module 10 is used to acquire the original text data and the first vector set in the vector database;

[0203] The processing module 20 is used to perform semantic vectorization processing on the original text data based on a preset semantic encoding model, and to standardize the generated initial vectors to obtain a second vector set.

[0204] The verification module 30 is used to perform semantic consistency verification between the second vector set and the first vector set, and to filter out each third vector whose semantics meet the preset requirements;

[0205] Embedding module 40 is used to embed each of the third vectors into a preset initial vector index map based on an improved incremental HNSW insertion strategy to obtain a first vector index map;

[0206] Optimization module 50 is used to optimize the first vector index graph by performing edge weight pruning and adjacent edge reconstruction based on the semantic similarity method, so as to obtain the second vector index graph.

[0207] Reconstruction module 60 is used to calculate the ratio of the total number of each third vector to the total number of vectors in the first vector set;

[0208] If the ratio reaches a preset reconstruction ratio threshold, nodes in the second vector index graph that satisfy the preset neighborhood condition with the semantic similarity to the third vector are extracted as centers to form a local subgraph. Semantic nearest neighbor connections are performed on the local subgraph to complete the reconstruction and obtain the updated vector database index.

[0209] For ease of description and brevity, the embodiments of the present invention include all the implementation methods in the above-described vector database index update method embodiments, which will not be repeated here.

[0210] Example 3:

[0211] This application provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the aforementioned index update method for a vector database.

[0212] The index update method for a vector database, if implemented as a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0213] Example 4

[0214] This embodiment provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements an index update method for any one of the vector databases as described in Embodiment 1.

[0215] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for index update of a vector database, characterized in that, The application relates to a method for updating a vector database index, and belongs to the technical field of information processing. The method comprises the following steps: acquiring original text data and a first vector set in a vector database; performing semantic vectorization processing on the original text data based on a preset semantic encoding model, normalizing the generated initial vectors, and obtaining a second vector set; performing semantic consistency verification on the second vector set and the first vector set, and screening out each third vector that meets preset requirements; embedding the third vectors into a preset initial vector index graph based on an improved incremental HNSW insertion strategy, and obtaining a first vector index graph; performing edge weight pruning and adjacent edge reconstruction optimization on the first vector index graph based on a semantic similarity method, and obtaining a second vector index graph; calculating the proportion of the total number of the third vectors to the total number of vectors in the first vector set; 2.The method of Claim 1, wherein, if the proportion reaches a preset reconstruction proportion threshold, extracting nodes in the second vector index graph that meet preset neighborhood conditions in terms of semantic similarity with the third vectors as the center, obtaining a local subgraph, performing semantic neighbor connection on the local subgraph to complete reconstruction, and obtaining an updated vector database index. The method comprises the following steps: splitting the title, the main text and the appendix content in the original text data, and cutting the main text according to a preset paragraph length to obtain a plurality of standardized text paragraphs; inputting the plurality of standardized text paragraphs into a preset semantic encoding model, so that the semantic encoding model encodes the semantic of each standardized text paragraph to generate corresponding initial vectors; wherein the semantic encoding model is a Sentence-BERT, MPNet or MiniLM model fine-tuned based on a preset field corpus; 3.The method of Claim 1, wherein, performing L2 norm normalization processing on each initial vector, and calculating the square root of each dimension component of each initial vector to obtain a second vector set. The method comprises the following steps: selecting a to-be-verified vector from the second vector set; calculating the semantic similarity between the to-be-verified vector and each vector in the first vector set; wherein the semantic similarity is obtained through vector dot product operation; selecting a first semantic similarity value with the maximum semantic similarity and comparing it with a preset semantic threshold; if the first semantic similarity value is smaller than the semantic threshold, it is determined that the to-be-verified vector meets the preset requirements, and the to-be-verified vector is taken as a third vector; 4.The method of Claim 1, wherein, until all vectors in the second vector set are verified, all to-be-verified vectors that meet the preset requirements are taken as the third vectors. The method comprises the following steps: based on a preset normalization coefficient, a preset attenuation coefficient and the maximum allowed layer number of the preset initial vector index graph, constructing an exponential distribution sampling function; The probability of each third vector being assigned to each level in the initial vector index graph is calculated by the exponential distribution sampling function, and the target insertion level of each third vector is determined according to the probability; Starting from the highest level of the initial vector index graph, the third vector is taken as a reference, the highest level candidate node set is obtained by searching for the highest semantic similarity candidate nodes in the existing nodes in the highest level, and then the next level is entered based on the highest level candidate node set to continue searching for candidate nodes that are more optimal than the third vector, and the process is repeated layer by layer until the bottom layer of the initial vector index graph is reached, and the bottom layer candidate node set is obtained; The top N candidate nodes in terms of semantic similarity are selected from the bottom layer candidate node set to form a neighbor node set; wherein N is a positive integer; A bidirectional connection edge is established between the third vector and each node in the neighbor node set, and the third vector is added to the corresponding target insertion level of the initial vector index graph, until all third vectors are embedded in the initial vector index graph, and a first vector index graph is obtained. 5.The method of Claim 1, wherein, The first vector index graph is optimized by edge weight pruning and adjacent edge reconstruction based on the semantic similarity method to obtain a second vector index graph, specifically as follows: All connection edges in the first vector index graph are traversed, the semantic similarity of the vectors corresponding to the nodes at both ends of each connection edge is calculated, and the semantic similarity is taken as the edge weight of the corresponding connection edge to obtain a first vector index graph with edge weight labels; Connection edges with edge weights less than a preset weak edge judgment threshold are marked as weak edges, and all weak edges are deleted from the first vector index graph with edge weight labels to obtain a first vector index graph after deleting weak edges; A node in the first vector index graph after deleting weak edges is selected as a current processing node, nodes with a semantic similarity not lower than the weak edge judgment threshold are filtered from the candidate neighbor set of the current processing node, the top M nodes are selected after sorting in descending order of semantic similarity, and a target neighbor node set of the current processing node is obtained; wherein M is a positive integer; and the candidate neighbor set is a node visited during the embedding of the third vector; Adjacent edges are established between the current processing node and the nodes in the target neighbor node set to obtain a local graph structure after single-node adjacent edge reconstruction; Until the adjacent edge reconstruction of all nodes in the first vector index graph after deleting weak edges is completed, all local graph structures after single-node adjacent edge reconstruction are summarized to obtain a second vector index graph. 6.The method of Claim 1, wherein, The proportion of the total number of third vectors to the total number of vectors in the first vector set is calculated; if the proportion reaches a preset reconstruction proportion threshold, a local subgraph is extracted from the second vector index graph with the third vector as the center, the nodes in the local subgraph that satisfy the preset neighborhood condition in terms of semantic similarity with the third vector are selected, and semantic neighbor connection is performed on the local subgraph to complete the reconstruction, and an updated vector database index is obtained, specifically as follows: The ratio of the total number of third vectors to the total number of vectors in the first vector set is calculated as an update ratio; The update ratio is compared with a preset reconstruction ratio threshold value, and if the update ratio value is greater than or equal to the preset reconstruction ratio threshold value, nodes in the second vector index graph are screened according to a preset neighborhood condition with each third vector as a center, the screened nodes are combined with the third vector, and a local subgraph corresponding to each third vector is obtained; A local subgraph is selected as a current processing subgraph, and original connection edges of all nodes in the current processing subgraph are deleted to obtain a local subgraph after the original edges are deleted; For each node in the local subgraph after the original edges are deleted, the semantic similarity of each node to other nodes in the subgraph is recalculated, and a preset number of nodes are selected as a target neighbor node set after being sorted from high to low according to the semantic similarity; A bidirectional connection edge is established between each node in the current processing subgraph and a node in the corresponding target neighbor node set to obtain a subgraph structure after the reconstruction of a single local subgraph is completed; Until the reconstruction of all local subgraphs is completed, all reconstructed local subgraphs are obtained. All reconstructed local subgraphs and the remaining part of the second vector index graph not included in the local subgraph are summarized to obtain an updated vector database index.

7. An index update apparatus of a vector database, characterized by comprising: Comprise: An acquisition module is configured to acquire original text data and a first vector set in a vector database; A processing module is configured to perform semantic vectorization processing on the original text data based on a preset semantic encoding model, and normalize the generated initial vectors to obtain a second vector set; A verification module is configured to perform semantic consistency verification on the second vector set and the first vector set, and screen out each third vector that meets a preset requirement; An embedding module is configured to embed each third vector into a preset initial vector index graph based on an improved incremental HNSW insertion strategy to obtain a first vector index graph; An optimization module is configured to perform edge weight pruning and adjacent edge reconstruction optimization on the first vector index graph based on a semantic similarity method to obtain a second vector index graph; A reconstruction module is configured to calculate a ratio of a total number of the third vectors to a total number of vectors in the first vector set; If the ratio reaches a preset reconstruction ratio threshold value, nodes in the second vector index graph that meet a preset neighborhood condition in terms of semantic similarity with the third vector are extracted to form a local subgraph, and semantic neighbor connection is performed on the local subgraph to complete reconstruction, thereby obtaining an updated vector database index.

8. The index update apparatus of a vector database according to claim 7, wherein, The original text data is subjected to semantic vectorization processing based on a preset semantic encoding model, and the generated initial vectors are normalized to obtain a second vector set, specifically as follows: The title, body and appendix content in the original text data are split, and the body is cut according to a preset paragraph length to obtain a plurality of standardized text segments; The plurality of standardized text segments are input into a preset semantic encoding model, so that the semantic encoding model encodes each standardized text segment to generate corresponding initial vectors; wherein the semantic encoding model is a Sentence-BERT, MPNet or MiniLM model fine-tuned on a preset field corpus. The L2 norm normalization processing is performed on each initial vector, and the square root of each dimension component of each initial vector is calculated to obtain a second vector set.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to perform the index updating method of the vector database according to any one of claims 1 to 6 when the computer program runs.

10. A terminal device, comprising: The computer readable storage medium comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the index updating method of the vector database according to any one of claims 1 to 6 when the computer program is executed.

Images