Block-level semantic difference increment updating system and method for vector database

Abstract

The invention relates to a block-level semantic difference increment updating system and method for a vector database, and belongs to the technical field of data processing, and the system comprises a client which is used for obtaining a new version and an old version of a document, and comparing block fingerprint sequences in the new version and the old version to obtain change metadata, the change metadata is sent to a cloud server, and the version type of the document is determined based on the timestamp; and the cloud server is used for querying the vector database based on the change metadata, obtaining an old fingerprint and an old vector of the affected block, and executing vectorization change and index change based on the old fingerprint and the old vector of the affected block so as to enable the updated vector database to be online based on a hot replacement mode. According to the system, the situation that the whole document is revectorized in each update is avoided, the delay from document modification to retrieval effectiveness is shortened by updating the vector database in real time, and the user experience is improved.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a block-level semantic differential incremental update system and method for vector databases. Background Technology

[0002] With the widespread application of Large Language Modeling (LLM), intelligent applications based on the RAG architecture, such as personal knowledge base question-answering systems, have become a core function of cloud note-taking services. The performance of such systems is highly dependent on the timeliness and accuracy of the underlying vectorized database. However, the current industry practice for maintaining vector databases is that the system triggers a complete update process whenever any modification is made to the source document (such as cloud notes). However, a complete update process leads to high resource consumption and update delays. Therefore, how to better implement vector database updates has become an urgent problem to be solved. Summary of the Invention

[0003] In view of the shortcomings of the prior art, the purpose of the invention is to provide a block-level semantic differential incremental update system and method for vector databases.

[0004] In a first aspect, the present invention proposes a block-level semantic differential incremental update system for a vector database, comprising: a client, configured to obtain a new version and an old version of a document, compare the block fingerprint sequences in the new version and the old version to obtain change metadata, and send the change metadata to a cloud server, wherein the version type of the document is determined based on a timestamp; the cloud server, configured to query the vector database based on the change metadata, obtain the old fingerprints and old vectors of the affected blocks, and perform vectorization changes and index changes based on the old fingerprints and old vectors of the affected blocks, so as to put the updated vector database online in a hot-swap manner.

[0005] Furthermore, both the new and old versions of the document include multiple text blocks, each of which corresponds to a unique fingerprint.

[0006] Further, by comparing the block fingerprint sequences in the new version and the old version, change metadata is obtained, including: comparing the first block fingerprint sequence of the new version and the second block fingerprint sequence of the old version to identify newly added blocks, deleted blocks, and modified blocks; and determining the change metadata based on the newly added blocks, the deleted blocks, and the modified blocks, wherein the change metadata includes document ID, change type, text block ID, and new text block fingerprint.

[0007] Further, the affected block includes the newly added block, the deleted block, and the modified block, wherein performing vectorization changes based on the old fingerprint and the old vector of the affected block includes: if the affected block is the newly added block, storing the new vector corresponding to the newly added block into the vector database; if the affected block is the deleted block, marking the vector corresponding to the deleted block as soft deletion or invalid.

[0008] Further, performing vectorization changes based on the old fingerprint and old vector of the affected block includes: if the affected block is the modified block, determining the semantic change type of the modified block; if the semantic change type is a significant semantic change, re-vectorizing and updating the corresponding vector record in the vector database; if the semantic change type is no significant semantic change, skipping vectorization and only updating the text cache of the modified block in the vector database.

[0009] Further, determining the semantic change type of the modified block includes: obtaining the new text content and the old text content of the modified block; calculating the similarity score between the new text content and the old text content based on a similarity calculation model; determining the semantic change type of the modified block based on the similarity score; wherein, if the similarity score is greater than a preset threshold, the semantic change type of the modified block is determined to be "no significant semantic change"; if the similarity score is not greater than the preset threshold, the semantic change type of the modified block is determined to be "significant semantic change".

[0010] Furthermore, the sub-indexes of the vector being changed are asynchronously reconstructed.

[0011] A second aspect of the present invention proposes a block-level semantic differential incremental update method for a vector database, comprising: obtaining a new version and an old version of a document, and comparing the block fingerprint sequences in the new version and the old version to obtain change metadata, wherein the version type of the document is determined based on a timestamp; querying the vector database based on the change metadata to obtain the old fingerprints and old vectors of the affected blocks, and performing vectorization changes and index changes based on the old fingerprints and old vectors of the affected blocks, so as to put the updated vector database online in a hot-swap manner.

[0012] A third aspect of the present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described in the second aspect of the present invention.

[0013] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the second aspect of the present invention.

[0014] The beneficial effects of this invention are as follows:

[0015] The present invention discloses a block-level semantic differential incremental update system and method for vector databases. The client is used to obtain new and old versions of a document, compare the block fingerprint sequences in the new and old versions to obtain change metadata, and send the change metadata to a cloud server. The document version type is determined based on a timestamp. The cloud server is used to query the vector database based on the change metadata, obtain the old fingerprints and old vectors of the affected blocks, and perform vectorization and index changes based on the old fingerprints and old vectors of the affected blocks to deploy the updated vector database online using a hot-swap method. This system avoids fully revectorizing the entire document with each update, shortens the delay from document modification to retrieval effectiveness by updating the vector database in real time, and improves the user experience. Attached Figure Description

[0016] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0017] Figure 1 This is a schematic diagram of a block-level semantic differential incremental update system for a vector database according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of a block-level semantic differential incremental update method for a vector database according to a specific embodiment of the present invention;

[0019] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0022] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0024] With the widespread application of Large Language Modeling (LLM), intelligent applications based on the RAG architecture, such as personal knowledge base question-answering systems, have become a core function of cloud note-taking services. The performance of such systems is highly dependent on the timeliness and accuracy of the underlying vectorized knowledge base.

[0025] Currently, the common practice in the industry for maintaining vector knowledge bases is as follows: when any modification is made to the source document (such as cloud notes), the system will trigger a complete update process, including: 1) re-parse and segment the entire document; 2) perform vectorization calculations on all text blocks; 3) insert all newly generated vectors into the vector database; 4) trigger the adjustment or global reconstruction of the vector index.

[0026] This method has the following inherent drawbacks: 1. Huge resource consumption: Even a minor modification to a single character in a document will lead to the entire document being revectorized. The computational resource consumption is proportional to the document size, resulting in a huge waste of cloud computing power and electricity, and high costs for service providers. 2. Significant update delay: The time required to rebuild the global vector index increases dramatically with the amount of data, causing users to wait a long time for their modified notes to take effect in the intelligent question answering system, failing to meet real-time requirements. 3. Poor system scalability: The extensive update model cannot support the "update storm" brought about by high-frequency editing operations from massive numbers of users, easily becoming a system performance bottleneck and limiting business expansion.

[0027] To this end, the present invention proposes a block-level semantic differential incremental update system, method, and related device for vector databases. Specifically, the block-level semantic differential incremental update system, method, and related device for vector databases according to embodiments of the present invention are described below with reference to the accompanying drawings.

[0028] Figure 1 This is a schematic diagram of a block-level semantic differential incremental update system for a vector database according to an embodiment of the present invention. It should be noted that the block-level semantic differential incremental update system for a vector database according to this embodiment can be applied to the block-level semantic differential incremental update method for a vector database according to this embodiment. This block-level semantic differential incremental update method for a vector database can be configured on an electronic device or in a server. This application does not limit the scope of this embodiment.

[0029] like Figure 1 As shown, the block-level semantic differential incremental update system for the vector database includes:

[0030] Client 110 is used to obtain the new and old versions of a document, compare the block fingerprint sequences in the new and old versions to obtain change metadata, and send the change metadata to the cloud server. The version type of the document is determined based on the timestamp.

[0031] Cloud server 120 is used to query the vector database based on the changed metadata, obtain the old fingerprints and old vectors of the affected blocks, and perform vectorization changes and index changes based on the old fingerprints and old vectors of the affected blocks, so as to put the updated vector database online in a hot-swap manner.

[0032] In other words, the block-level semantic differential incremental update system of this vector database is a client-cloud collaborative intelligent document processing system used to achieve real-time detection of document changes and seamless synchronization of vectorized knowledge. The system consists of a client (110) and a cloud server (120), jointly completing the entire process from document change detection to hot updates of the vector database. This system avoids fully re-vectorizing the entire document with each update, and by updating the vector database in real time, it shortens the delay from document modification to retrieval effectiveness, thus improving the user experience.

[0033] In embodiments of the present invention, the version type of a document is determined based on a timestamp. For example, if the timestamps of a document include 10:17 AM on March 5, 2018 and 3:20 PM on March 5, 2018, the document at 3:20 PM on March 5, 2018 can be considered the new version, and the document at 10:17 AM on March 5, 2018 can be considered the old version.

[0034] In embodiments of the present invention, both the new and old versions of the document include multiple text blocks, each text block corresponding to a unique fingerprint. This leads to the generation of the block fingerprint sequence for the new version and the block fingerprint sequence for the old version (i.e., the first block fingerprint sequence of the new version and the second block fingerprint sequence of the old version).

[0035] The unique identifier (i.e., unique fingerprint) of a text block can be calculated using a hash algorithm (such as SimHash or MD5).

[0036] In an embodiment of the present invention, when a new version and an old version of a document are obtained, the first fingerprint sequence of the new version and the second fingerprint sequence of the old version are compared to identify the added block, the deleted block, and the modified block; based on the added block, the deleted block, and the modified block, the change metadata is determined, wherein the change metadata includes the document ID, the change type, the text block ID, and the new fingerprint of the text block.

[0037] The new fingerprint of a text block can be understood as the fingerprint corresponding to the text block in the new version.

[0038] In an embodiment of the present invention, if the changed metadata is determined, the changed metadata can be uploaded to the cloud server 120.

[0039] In an embodiment of the present invention, when the cloud server 120 receives the change metadata, it can query the vector database based on the change metadata to obtain the old fingerprint and old vector of the affected block.

[0040] The affected blocks include newly added blocks, deleted blocks, and modified blocks.

[0041] In an embodiment of the present invention, when the affected block is a newly added block, the new vector corresponding to the newly added block is stored in the vector database.

[0042] In other words, if the affected block is a newly added block, a new vector ID can be generated for the newly added block and stored in the vector database to perform a complete vectorization operation.

[0043] In an embodiment of the present invention, when the affected block is a deletion block, the vector corresponding to the deletion block is marked as soft deletion or invalid.

[0044] Soft deletion or invalidation can be understood as non-physical deletion, which facilitates version rollback.

[0045] In an embodiment of the present invention, when the affected block is a modified block, the semantic change type of the modified block is determined; when the semantic change type is a significant semantic change, it is re-vectorized and the corresponding vector record in the vector database is updated; when the semantic change type is no significant semantic change, vectorization is skipped and only the text cache of the modified block in the vector database is updated.

[0046] In other words, when the semantic change type is a major semantic change, it is re-vectorized (fully calculated), and the vector record corresponding to the modified block ID in the vector database is updated. The old vector can be archived or marked as a historical version. When the semantic change type is no major semantic change, vectorization is skipped to avoid duplicate calculations. Only the text cache of the modified block in the vector database is updated, keeping the original vector ID unchanged. The original vector ID is still used during retrieval.

[0047] For example, significant semantic changes include, but are not limited to, paragraph rewriting and the addition of important arguments. No significant semantic changes include, but are not limited to, punctuation corrections, synonym replacements, and minor spelling corrections.

[0048] Specifically, the process involves obtaining the new and old text content of the modified block; calculating the similarity score between the new and old text content based on a similarity calculation model; and determining the semantic change type of the modified block based on the similarity score.

[0049] Among them, if the similarity score is greater than a preset threshold, the semantic change type of the modified block is determined to be no significant semantic change;

[0050] Specifically, if the similarity score is not greater than a preset threshold, the semantic change type of the modified block is determined to be a significant semantic change.

[0051] The similarity calculation model can be a sentence vector model such as Sentence-BERT.

[0052] In embodiments of the present invention, to accelerate vector similarity search, the vector database establishes an index (such as HNSW, IVF, etc.) for the vector set. When the vector data changes, the index also needs to be updated to reflect the latest data. However, rebuilding the entire index (global index) is very time-consuming, especially when the number of vectors is large. Therefore, in embodiments of the present invention, the sub-index of the vector that has been changed is rebuilt asynchronously.

[0053] In this context, a sub-index can be understood as an index being divided into multiple parts (e.g., by document, by user, or by a certain partitioning strategy), so that when only a part of the vector changes, only the affected part of the index needs to be rebuilt.

[0054] In embodiments of the present invention, the specific implementation of asynchronous reconstruction of the sub-indexes of the changed vector includes:

[0055] 1. Determine the scope of the change's impact: Based on the vector IDs of this change, determine which sub-index(s) these vectors belong to. For example, if each document has a sub-index, then the sub-indexes corresponding to the documents affected by this change need to be rebuilt.

[0056] 2. Notify the vector database: The incremental update engine sends a notification to the vector database, informing it of the information of the sub-indexes that need to be rebuilt (e.g., the identifier of the sub-index).

[0057] 3. Asynchronous sub-index rebuilding: Vector Database rebuilds affected sub-indexes asynchronously in the background. Asynchronous means that the rebuild task does not block the current update operation; the engine can return a response immediately, while the rebuild task runs in the background.

[0058] 4. Reconstruction Process: The vector database reads the latest vector data (including newly added, deleted, and / or modified vectors) corresponding to the sub-index and then rebuilds the sub-index. When rebuilding the sub-index, the original index structure (such as an HNSW graph) can be used, or an incremental update algorithm can be employed.

[0059] 5. Hot switching: After the reconstruction is completed, the old sub-index is replaced with the new sub-index. This process is transparent to queries, that is, query requests will be routed to the new sub-index.

[0060] By asynchronously rebuilding the sub-indexes of the changed vector, only a portion of the index is rebuilt, reducing computational load and time. Asynchronous rebuilding does not block other operations, allowing the system to continue processing other requests. As the data volume grows, the sub-indexes can be distributed across different nodes, achieving distributed index rebuilding. This improves system performance and availability while maintaining data consistency and query accuracy.

[0061] In an embodiment of the present invention, after the new sub-index is rebuilt, the vector database brings it online in a hot-swap manner, seamlessly replacing the old sub-index. This process is unaffected and uninterrupted by front-end retrieval requests.

[0062] In one specific embodiment of the present invention, suppose a user modifies a sentence in their "Project Plan" document: Original content (block ID: 205): "The mid-term project review meeting will be held next week." Modified content: "The mid-term project review meeting will be held next Tuesday."

[0063] The client calculates the new fingerprint of the modified block and compares it with the old fingerprint stored locally, finding that the content of block ID:205 has changed. Then, the change metadata {note_id:123, chunk_id:205, change_type:modify, new_fingerprint:xyz...} is uploaded to the cloud server. The cloud server receives the change metadata and retrieves the old text and fingerprint of block 205. It calculates the semantic similarity between "next week" and "next Tuesday" using a sentence vector model, and the score is extremely high (e.g., greater than 0.98), indicating no significant semantic change. Therefore, it does not trigger re-vectorization of the block, but only updates the text cache of the block in the vector database with the new text. Since the vector remains unchanged, there is no need to rebuild its sub-index. Thus, whether searching for "next week's meeting" or "next Tuesday's meeting," the block can be correctly located because the text cache has been updated. However, the entire update process avoids unnecessary vector calculations and index rebuilding, saving significant resources. If the user changes the sentence to "The mid-term review meeting for the project has been cancelled", the semantic similarity will be extremely low. The engine will determine this as a major semantic change, and will then perform re-vectorization and update the sub-index.

[0064] The block-level semantic differential incremental update system for vector databases according to embodiments of the present invention, by determining the semantic change type of the modified block, skips vectorization when the semantic change type is no significant semantic change, minimizing the computational overhead of incremental updates, reducing unnecessary vectorization calculations, and lowering cloud service costs. By asynchronously rebuilding the sub-index of the changed vector, global index rebuilding is avoided, and the local rebuilding speed of the sub-index is extremely fast, enabling the vector database to respond to changes within seconds, greatly improving the user experience. The system's ability to handle incremental updates is no longer limited by the total data volume, but only by the amount of changed data per unit time, allowing it to easily handle massive numbers of users and high-frequency editing scenarios, enhancing system stability. Furthermore, by updating the text cache, the system ensures the consistency between the text accuracy of the search results and the vector representation, solving the problem of missing latest text information that may result from solely relying on vectors.

[0065] In embodiments of the present invention, a block-level semantic differential incremental update method for vector databases is also proposed, such as... Figure 2 As shown, the block-level semantic differential incremental update method for this vector database includes:

[0066] S210: Obtain the new and old versions of the document, and compare the block fingerprint sequences in the new and old versions to obtain change metadata, wherein the version type of the document is determined based on the timestamp.

[0067] In embodiments of the present invention, both the new and old versions of the document include multiple text blocks, each text block corresponding to a unique fingerprint.

[0068] In an embodiment of the present invention, the first fingerprint sequence of the new version and the second fingerprint sequence of the old version are compared to identify the newly added block, the deleted block, and the modified block; based on the newly added block, the deleted block, and the modified block, the change metadata is determined, wherein the change metadata includes document ID, change type, text block ID, and new fingerprint of text block.

[0069] S220 queries the vector database based on the changed metadata to obtain the old fingerprints and old vectors of the affected blocks, and performs vectorization changes and index changes based on the old fingerprints and old vectors of the affected blocks, so as to put the updated vector database online in a hot-swap manner.

[0070] In embodiments of the present invention, the affected blocks include added blocks, deleted blocks, and modified blocks.

[0071] In an embodiment of the present invention, when the affected block is a newly added block, the new vector corresponding to the newly added block is stored in the vector database; when the affected block is a deleted block, the vector corresponding to the deleted block is marked as soft deletion or invalid.

[0072] In an embodiment of the present invention, when the affected block is a modified block, the semantic change type of the modified block is determined; when the semantic change type is a significant semantic change, it is re-vectorized and the corresponding vector record in the vector database is updated; when the semantic change type is no significant semantic change, vectorization is skipped and only the text cache of the modified block in the vector database is updated.

[0073] The process involves: acquiring the new and old text content of the modified block; calculating the similarity score between the new and old text content based on a similarity calculation model; determining the semantic change type of the modified block based on the similarity score; and determining the semantic change type of the modified block as follows: if the similarity score is greater than a preset threshold, the semantic change type of the modified block is determined to be "no significant semantic change"; if the similarity score is not greater than the preset threshold, the semantic change type of the modified block is determined to be "significant semantic change".

[0074] In an embodiment of the present invention, the sub-index of the changed vector is asynchronously reconstructed.

[0075] According to an embodiment of the present invention, a block-level semantic differential incremental update method for a vector database obtains a new version and an old version of a document, compares the block fingerprint sequences in the new and old versions to obtain change metadata, wherein the document version type is determined based on a timestamp; based on the change metadata, the vector database is queried to obtain the old fingerprints and old vectors of the affected blocks, and vectorization and index changes are performed based on the old fingerprints and old vectors of the affected blocks to deploy the updated vector database online using a hot-swap approach. This method avoids fully revectorizing the entire document with each update, shortens the delay from document modification to retrieval effectiveness by updating the vector database in real time, and improves the user experience.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0077] According to one aspect of the present invention, an electronic device is provided.

[0078] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 3 As shown, an electronic device may include one or more ( Figure 3 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the electronic device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the terminal device described above. For example, the terminal device may also include components that are more... Figure 3 The more or fewer components shown, or having the same Figure 3 Equivalent functions or ratios shown Figure 3 The functions shown have more different configurations.

[0079] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the block-level semantic differential incremental update method for vector databases in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to terminal devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the switching device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0081] This invention proposes a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute a block-level semantic differential incremental update method for a vector database.

[0082] The applicant of this invention has provided a detailed description of the embodiments of the invention in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred embodiments of the invention. The detailed description is only intended to help readers better understand the spirit of the invention and is not intended to limit the scope of protection of the invention. On the contrary, any improvements or modifications made based on the inventive spirit of the invention should fall within the scope of protection of the invention.

[0083] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0084] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A block-level semantic differential incremental update system for a vector database, characterized in that, include: A client is used to obtain new and old versions of a document, compare the block fingerprint sequences in the new and old versions to obtain change metadata, and send the change metadata to a cloud server, wherein the version type of the document is determined based on a timestamp; The cloud server is used to query the vector database based on the changed metadata, obtain the old fingerprints and old vectors of the affected blocks, and perform vectorization changes and index changes based on the old fingerprints and old vectors of the affected blocks, so as to put the updated vector database online in a hot-swap manner. 2.The block-level semantic differential incremental update system of vector database according to claim 1, characterized in that, Both the new and old versions of the document include multiple text blocks, each of which corresponds to a unique fingerprint. 3.The block-level semantic differential incremental update system of vector database according to claim 1, characterized in that, By comparing the block fingerprint sequences in the new version and the old version, change metadata is obtained, including: The first fingerprint sequence of the new version is compared with the second fingerprint sequence of the old version to identify the added block, deleted block and modified block; Based on the newly added block, the deleted block, and the modified block, the change metadata is determined, wherein the change metadata includes document ID, change type, text block ID, and new fingerprint of text block. 4.The block-level semantic differential incremental update system of vector database according to claim 3, characterized in that, The affected blocks include the newly added block, the deleted block, and the modified block, wherein performing vectorized changes based on the old fingerprint and the old vector of the affected blocks includes: If the affected block is the newly added block, the new vector corresponding to the newly added block is stored in the vector database; If the affected block is the deleted block, the vector corresponding to the deleted block is marked as soft deletion or invalid. 5.The block-level semantic differential incremental update system of vector database according to claim 4, characterized in that, Performing vectorization changes based on the old fingerprint and old vector of the affected block includes: If the affected block is the modified block, determine the semantic change type of the modified block; If the semantic change type is a significant semantic change, the vector is revectorized and the corresponding vector record in the vector database is updated. If the semantic change type is no significant semantic change, vectorization is skipped, and only the text cache of the modified block in the vector database is updated.

6. The block-level semantic differential incremental update system for vector databases according to claim 5, characterized in that, Determining the semantic change type of the modified block includes: Obtain the new and old text content of the modified block; Based on the similarity calculation model, the similarity score between the new text content and the old text content is calculated; Based on the similarity score, the semantic change type of the modified block is determined; Wherein, if the similarity score is greater than a preset threshold, the semantic change type of the modified block is determined to be "no significant semantic change"; If the similarity score is not greater than the preset threshold, the semantic change type of the modified block is determined to be the semantic major change.

7. The block-level semantic differential incremental update system for vector databases according to claim 4, characterized in that, The sub-indexes of the vector being changed are reconstructed asynchronously.

8. A block-level semantic differential incremental update method for vector databases, characterized in that, include: Obtain the new and old versions of the document, and compare the block fingerprint sequences in the new and old versions to obtain change metadata, wherein the version type of the document is determined based on the timestamp; Based on the changed metadata, the vector database is queried to obtain the old fingerprints and old vectors of the affected blocks. Vectorization changes and index changes are then performed based on the old fingerprints and old vectors of the affected blocks to bring the updated vector database online using a hot-swap method.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of claim 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to claim 8.

Images