Vector data self-adaptive updating method and device, equipment and medium

Abstract

The invention relates to the technical field of data analysis, and discloses a vector data adaptive updating method and device, equipment and a medium, and the method comprises the steps: extracting a feature fingerprint of target vector data stored in a preset vector library, and recognizing an influence region of the feature fingerprint in the vector library through a multi-level index structure of the vector library; determining a self-adaptive updating strategy of the target vector data in the vector library according to the influence area and a preset updating trigger event of the vector library; and updating the target vector data into a vector library through a self-adaptive updating strategy, and updating the multi-level index structure based on index data corresponding to the updated target vector data in the vector library. And the efficiency of adaptive updating of the vector data is improved.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to an adaptive update method, apparatus, device, and medium for vector data. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, vector data processing has become a core technology in many application fields. However, vector retrieval systems face significant challenges when processing massive amounts of data. Traditional vector update methods mainly adopt a full reconstruction strategy, which requires rebuilding the entire vector index structure every time data changes. This approach is clearly insufficient when processing large-scale dynamic data.

[0003] Existing vector library update technologies are geographic feature-level update operations, but their update mechanisms are mainly designed for specific application scenarios, lacking versatility and scalability. Furthermore, data updates primarily rely on full reconstruction, which triggers global reconstruction when there are local changes, resulting in low efficiency when adaptively updating vector data. Summary of the Invention

[0004] This invention provides an adaptive update method, apparatus, device, and medium for vector data to solve the technical problem of low efficiency in adaptive updates of vector data.

[0005] Firstly, an adaptive update method for vector data is provided, including: Extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence area of ​​the feature fingerprint in the vector library through the multi-level index structure of the vector library; The adaptive update strategy for the target vector data in the vector library is determined based on the affected area and the preset update triggering events of the vector library. The target vector data is updated to the vector library using the adaptive update strategy, and the multi-level index structure is updated based on the index data corresponding to the updated target vector data in the vector library.

[0006] Secondly, an adaptive update device for vector data is provided, comprising: The influence region identification module is used to extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence region of the feature fingerprint in the vector library through the multi-level index structure of the vector library; An adaptive update strategy determination module is used to determine the adaptive update strategy of the target vector data in the vector library based on the affected area and the preset update triggering event of the vector library; The multi-level index structure update module is used to update the target vector data to the vector library through the adaptive update strategy, and update the multi-level index structure based on the index data corresponding to the updated target vector data in the vector library.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the adaptive update method for the vector data described above.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the adaptive update method for the vector data described above.

[0009] In the aforementioned adaptive update method, apparatus, device, and medium for vector data, the feature fingerprint of the target vector data stored in a preset vector library can be extracted by the client. The influence area of ​​the feature fingerprint within the vector library is identified through its multi-level index structure. An adaptive update strategy for the target vector data in the vector library is determined based on the influence area and preset update trigger events. The target vector data is updated to the vector library using the adaptive update strategy, and the multi-level index structure is updated based on the index data corresponding to the updated target vector data in the vector library. The updated multi-level index structure is then fed back to the client. In this invention, the optimal update strategy is adaptively selected through the multi-level index structure. The multi-level index structure includes a global index layer, a regional index layer, and a local index layer. The global index layer maintains coarse-grained information about the overall data distribution, the regional index layer manages medium-grained data clustering information, and the local index layer processes fine-grained vector proximity relationships to support efficient incremental updates. The optimal update strategy is adaptively selected based on the changing characteristics of the vector data and the system load, significantly reducing the frequency of full reconstruction while ensuring update accuracy, thereby solving the technical problem of low efficiency in adaptive updates of vector data. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for an adaptive update method for vector data according to an embodiment of the present invention; Figure 2This is a flowchart illustrating an adaptive update method for vector data in one embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S1; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method of step S2; Figure 5 This is a schematic diagram of an adaptive update device for vector data in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The adaptive update method for vector data provided in this embodiment of the invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can extract the feature fingerprints of the target vector data stored in a preset vector library through the client, identify the influence area of ​​the feature fingerprints in the vector library through the multi-level index structure of the vector library, determine the adaptive update strategy of the target vector data in the vector library based on the influence area and the preset update trigger event of the vector library, update the target vector data to the vector library through the adaptive update strategy, and update the multi-level index structure based on the index data corresponding to the updated target vector data in the vector library, and feed the updated multi-level index structure back to the client. In this invention, the optimal update strategy is adaptively selected through the multi-level index structure, which includes a global index layer, a regional index layer, and a local index layer. The global index layer maintains coarse-grained information of the overall data distribution, the regional index layer manages medium-grained data clustering information, and the local index layer processes fine-grained vector proximity relationships to support efficient incremental updates. Based on the change characteristics of the vector data and the system load, the optimal update strategy is adaptively selected, which significantly reduces the frequency of full reconstruction while ensuring update accuracy, thereby solving the technical problem of low efficiency in adaptive updates of vector data. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the adaptive update method for vector data provided in this embodiment of the invention includes the following steps: S1. Extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence area of ​​the feature fingerprint in the vector library through the multi-level index structure of the vector library.

[0015] In this embodiment of the invention, the target vector data refers to a new vector used to update the vector library. The feature fingerprint contains a unique identifier and metadata description, and is a digital label used to uniquely identify and describe the target vector data, similar to the uniqueness and identifiability of human fingerprints.

[0016] In this embodiment of the invention, reference is made to Figure 3 As shown, the step of extracting the feature fingerprint of the target vector data stored in the preset vector library includes: S31. Convert the data type of the target vector data stored in the preset vector library; S32. Calculate the hash value corresponding to the target vector data after data type conversion; S33. Identify the basic metadata and business metadata corresponding to the target vector data; S34. Encapsulate the basic metadata, the business metadata, and the hash value into the feature fingerprint of the target vector data.

[0017] In detail, since vector data exists in various forms, it is necessary to convert the corresponding data types of the vector data, such as converting the elements to floating-point types and unifying the precision to avoid inconsistent hash values ​​caused by format differences. Then, a unique fingerprint is generated for the vector through a hash algorithm for fast deduplication and index association.

[0018] Specifically, to extract the background information of the vector, the basic metadata includes the number of features of the vector, the timestamp of the vector, and the data source of the vector; the business metadata includes image scene and text scene, so the hash value and metadata are integrated into a unified format for easy storage and transmission. That is, the original vector is stored in the vector database by establishing a relationship through the hash value, thereby obtaining the feature fingerprint corresponding to the target vector data.

[0019] Furthermore, to avoid triggering a global rebuild due to local changes, and to maintain consistency between different levels through an incremental synchronization mechanism to ensure the accuracy of query results, a multi-level index structure is constructed so that the appropriate index layer can be selected for updating based on the scope of the update operation.

[0020] In this embodiment of the invention, the multi-level index structure consists of a global index layer, a regional index layer, and a local index layer. The global index layer maintains coarse-grained information about the overall data distribution, the regional index layer manages medium-grained data clustering information, and the local index layer processes fine-grained vector proximity relationships.

[0021] In this embodiment of the invention, before identifying the influence region of the feature fingerprint within the vector library through the multi-level index structure of the vector library, the method further includes: Perform cluster analysis on all vectors in the vector library to obtain global clusters, and determine the global index layer corresponding to the vector library based on the global clusters. The global cluster is divided into regional clusters according to the attribute type corresponding to the global cluster, and the regional index layer corresponding to the vector library is determined according to the regional clusters. Extract the vector data and vector relationships corresponding to the region clusters, and construct an index file for the region clusters based on the vector data and vector relationships; The region clusters are divided into local clusters according to the index file, and the local index layer corresponding to the vector library is determined based on the local clusters. The vector library is constructed using the global index layer, the regional index layer, and the local index layer, forming a multi-level index structure.

[0022] In detail, cluster analysis is performed on all vectors in the vector library to obtain several global clusters. A global index layer is then determined based on these global clusters. That is, using global clustering algorithms (such as K-means or hierarchical clustering), the entire vector space is divided into several macro-clusters. Each cluster stores the centroid vector, cluster ID, and regional index layer entry pointer, and stores them in a Redis hash table with the cluster ID as the key. If the vector library contains user behavior vectors and product feature vectors, global clustering may divide them into user vector clusters and product vector clusters. The global index layer, as the highest level of the index structure, is used to quickly locate the macro-category to which the data belongs. For example, the global index can directly determine whether the target vector belongs to the user cluster or the product cluster, avoiding traversing the entire vector library.

[0023] Specifically, the global cluster is further divided into regional clusters based on the attribute types corresponding to the global clusters. Each global cluster is divided into several regional clusters, which are then stored using a BIRCH tree. Each regional cluster records the minimum bounding sphere parameters, such as the centroid, radius, and number of vectors, and stores them in a MySQL database, thus obtaining the regional index layer. Furthermore, each regional cluster uses an HNSW index to store vector IDs and nearest neighbor relationships. Using a Milvus vector database, each regional cluster corresponds to a Collection containing vector data and index files. This constructs a multi-level index structure consisting of the global index layer, regional index layer, and local index layer. As the data scale grows, the index levels can be increased or the granularity of each level can be adjusted to adapt to changes without requiring major modifications to the overall architecture.

[0024] Furthermore, by calculating the similarity distribution between the new vector and the existing vector set, the scope of influence is determined through cluster analysis, and then updates are performed using local reconstruction or incremental insertion.

[0025] In this embodiment of the invention, the affected region refers to the index range or physical storage area corresponding to the set of vectors in the vector library that are highly correlated with the target vector data. When the target vector data is updated, the vectors in this region will be affected by similarity or correlation and need to be updated accordingly, such as fusion or reconstruction.

[0026] In this embodiment of the invention, identifying the influence region of the feature fingerprint within the vector library through the multi-level index structure of the vector library includes: The feature fingerprint is matched with the cluster type corresponding to the global cluster in the global index layer to obtain the global candidate cluster corresponding to the feature fingerprint; Calculate the global similarity between the target vector data corresponding to the feature fingerprint and the cluster centroid of the global candidate cluster, and select the global candidate cluster with the largest global similarity as the target global cluster; Identify the region cluster corresponding to the global cluster based on the region index pointer corresponding to the target global cluster; Calculate the boundary distance between the target vector data and the region cluster, and filter the target region clusters in the region clusters based on the boundary distance; Identify the local index layer corresponding to the target vector data based on the index file corresponding to the target region cluster, and calculate the local similarity between the target vector data and each vector data in the local index layer; When the local similarity is greater than a preset similarity threshold, the vector data in the local index layer is used as the target vector data, and the influence area of ​​the feature fingerprint in the vector library is determined according to the index range of the target vector data.

[0027] In detail, the feature fingerprint is matched with the cluster type of each global cluster in the global index layer to filter out global candidate clusters that meet the conditions. The cluster type includes macro attributes such as data type and business category. The global similarity between the target vector data and the cluster centroid of each global candidate cluster is calculated, such as cosine similarity and Euclidean distance. The cluster with the highest similarity is selected as the target global cluster. The cluster centroid is the center vector of the vectors in the cluster and represents the typical features of the cluster. The coarse-grained filtering of the global index layer quickly narrows the search range, avoids full database traversal, and improves query efficiency.

[0028] Specifically, the region index pointer refers to the region cluster index under the global cluster. Based on the region index pointer stored in the target global cluster, the corresponding set of region clusters is found, and the boundary distance between the target vector data and each region cluster is calculated. Region clusters with a distance less than a threshold are selected as target region clusters. The boundary distance refers to the distance from the spatial boundary of the cluster to the target vector. Then, the range is further refined within the global cluster through the region index layer. The area of ​​influence is narrowed based on spatial location or attribute boundary to improve matching accuracy.

[0029] Furthermore, in the similarity calculation stage, an improved hierarchical relevance propagation method is adopted, which can more accurately identify the semantic associations between vectors. For high-dimensional vector data, the algorithm introduces a dimension reduction projection technique to reduce computational complexity while maintaining the essential characteristics of the data.

[0030] In this embodiment of the invention, the boundary distance refers to the spatial boundary between the target vector data and the region cluster, such as the distance between the minimum bounding sphere (MBR) and the hyperrectangular boundary.

[0031] In this embodiment of the invention, the step of filtering the target region clusters in the region clusters based on the boundary distance includes: When the boundary distance is less than a preset distance threshold, the region cluster is designated as the first region cluster. Identify whether the density of vector data in the first region cluster exceeds a preset density threshold; When the density of vector data in the first region cluster exceeds the density threshold, the first region cluster is taken as the target region cluster.

[0032] When the density of vector data in the first region cluster does not exceed the density threshold, the parent region cluster corresponding to the first region cluster is identified. The target region cluster within the region cluster is determined based on the parent region cluster.

[0033] In detail, the boundary distances between the target vector and all region clusters are calculated. Region clusters with distances less than a preset distance threshold are retained and denoted as the first region cluster. For each first region cluster, the vector density is calculated, and it is determined whether it exceeds the density threshold. If the density of the vector data exceeds the density threshold, it is in a high-density cluster and can be directly used as the target region cluster without further expansion. The boundary division of high-density clusters is usually more reliable, and the boundary of the target vector is considered to be within the influence range of the cluster. If the density of the vector data does not exceed the density threshold, the parent cluster of the first region cluster is queried, such as the global cluster to which the region cluster belongs or a higher-level region cluster. Among all the child clusters contained in the parent cluster, region clusters with distances less than the threshold are re-selected as new target region clusters. Low-density clusters may have unreliable boundary distance calculations due to data sparsity, so the selection range needs to be expanded through the parent cluster to avoid missing neighboring clusters. The number of vectors and spatial volume within the cluster can be obtained in real time through the metadata of the region index layer, and the density threshold is dynamically adjusted according to the average density of the global cluster.

[0034] Specifically, the corresponding local index layer is located based on the index file of the target region cluster. The index file records the index structure and vector relationships of vector data within the cluster. Then, the local similarity between the target vector data and each vector data in the local index layer is calculated. When the similarity exceeds a preset threshold, these vector data are marked as target vector data. Based on the index range of the target vector data, such as the position interval or pointer set in the local index, the influence area of ​​the feature fingerprint in the vector library is determined. Precise matching is performed in the local index layer. The set of vectors that are highly related to the target vector is filtered out through the similarity threshold. Finally, the specific range affected by the update is determined.

[0035] Furthermore, based on the changing characteristics of the vector data and the system load, the optimal update strategy is adaptively selected, which significantly reduces the frequency of full reconstruction while ensuring update accuracy.

[0036] S2. Determine the adaptive update strategy of the target vector data in the vector library based on the affected area and the preset update trigger event of the vector library.

[0037] In this embodiment of the invention, the adaptive update strategy is an intelligent mechanism by which the vector database dynamically selects the optimal update method and timing based on the impact of data changes and multidimensional environmental factors. The update triggering events include system load, data change frequency, and data query performance.

[0038] In this embodiment of the invention, reference is made to Figure 4 As shown, the step of determining the adaptive update strategy for the target vector data in the vector library based on the affected area and the preset update trigger events of the vector library includes: S41. Calculate the target distance between the target vector data and the cluster centers of the affected region, and determine the influence degree of the affected region based on the target distance; S42. Determine the update strategy for the target vector data based on the influence degree and the preset influence threshold; S43. Determine the adaptive update strategy for the target vector data in the vector library through the preset update trigger event and the update strategy.

[0039] In detail, multiple environment- or business-related update trigger events are predefined. Real-time feature values ​​corresponding to each factor dimension are obtained based on the system's operating status, such as current CPU utilization, data existence time, and number of regional queries. The impact reflects the degree of disturbance of the target vector to the existing clustering structure. That is, the impact of the update on the overall index structure is evaluated by calculating the distance between the new vector and the existing cluster centers. Based on the impact and the impact threshold, the update strategy for the target vector data is determined. When the impact is lower than the preset threshold, a local incremental update is adopted; when the impact exceeds the threshold but is lower than the reconstruction threshold, a regional reconstruction is adopted; and only when the impact exceeds the reconstruction threshold is a full reconstruction operation triggered.

[0040] Specifically, if the system load is low, the update trigger event is a real-time update; if the load is high, the update trigger event is a delayed update. If the update strategy is a local enhancement update and the update trigger event is a real-time update, then a local reconstruction is executed immediately. If the update strategy is a local enhancement update and the update trigger event is a delayed update, then the update request is queued and processed in batches when the system load decreases, reducing real-time resource contention. If the update strategy is a global reconstruction update and the update trigger event is a real-time update, then a background thread is started to perform global index reconstruction, such as re-dividing cluster centers, without blocking front-end requests. If the update strategy is a global reconstruction update and the update trigger event is a delayed update, then only incremental insertion is performed, the global index structure is not adjusted, and it is marked as pending optimization. Thus, an adaptive update strategy for the target vector data can be generated based on different update trigger events and update strategies. Through the adaptive mechanism, the vector database can minimize update costs while ensuring query accuracy, achieving intelligent allocation of system resources.

[0041] Furthermore, to ensure the adaptability of the update strategy, a parameter optimization mechanism is used to automatically adjust key parameters according to the actual application scenario and data characteristics. The parameter tuning method based on reinforcement learning optimizes key configurations such as update thresholds and clustering parameters by continuously monitoring system performance indicators. It also integrates online learning capabilities, which can learn the optimal strategy from historical update operations and apply the learning results to subsequent update decisions. This self-learning mechanism enables the algorithm to adapt to different types of data distributions and application scenarios.

[0042] S3. Update the target vector data to the vector library using the adaptive update strategy, and update the multi-level index structure based on the index data corresponding to the updated target vector data in the vector library.

[0043] In this embodiment of the invention, the automatic switching strategy based on the degree of impact not only ensures the efficiency of high-frequency, small-amplitude updates, but also copes with structural adjustments to disruptive data, realizing the transformation from fine-grained local adjustments to global structural reorganization, covering the data update needs of all scenarios.

[0044] In this embodiment of the invention, updating the target vector data to the vector library using the adaptive update strategy includes: When the adaptive update strategy is a local incremental update strategy, the vector data in the influence range is fused to obtain fused vector data; The fused vector data is incrementally inserted into the local index layer corresponding to the vector library; When the adaptive update strategy is a regional reconstruction strategy, the region cluster corresponding to the target vector data is identified, the region cluster is reconstructed according to the target vector data, and the regional index position of the target vector data in the vector library is determined according to the reconstructed region cluster. Update the target vector data to the corresponding region index layer of the vector library according to the region index position; When the adaptive update strategy is a full reconstruction strategy, the target global cluster corresponding to the target vector data is identified, the target vector data is updated in the target global cluster according to the preset initial position, and the global cluster corresponding to the vector library is reconstructed according to the updated target global cluster. The global index position of the target vector data in the vector library is determined based on the reconstructed global cluster, and the target vector data is updated to the global index layer corresponding to the vector library according to the global index position.

[0045] In detail, when the target vector is highly similar to an existing cluster, incremental updates are performed only within a local area to avoid affecting the global structure. This involves feature fusion between the target vector and existing vectors within the influence range to reduce the disturbance of a single sample to the cluster structure. The similarity (e.g., cosine similarity) between the target vector and each vector within the influence range is calculated and used as the weight for weighted fusion. The fused vector is then inserted into the local index layer, and the metadata of the local cluster, such as centroid and vector count, is updated. This allows for fine-tuning of the local cluster centroid without requiring index restructuring, maintaining stable query performance. When the target vector differs significantly from existing clusters but does not form a new category, the structure of the target region cluster is restructured to adapt to the distribution of new data. This involves locating the region cluster to which the target vector belongs through multi-level indexing. For example, adding a vaporwave electron vector locates the regular electron region cluster under the global electronic music cluster. If the distance between the target vector and the centroid of the region cluster exceeds a threshold or the cluster density is below a threshold, the region cluster is divided into two sub-clusters, such as dividing regular electrons into regular electrons and experimental electrons. Alternatively, clustering algorithms (DBSCAN or K-means) can be used. The vectors (including the target vector) within the region cluster are re-clustered, and then the old region cluster index is deleted, the index of the new sub-cluster is created, and the metadata of the region index layer is updated.

[0046] Specifically, when the target vector represents a completely new category or has a significant impact on the global structure, the index structure of the entire vector library is reconstructed. If the target vector cannot match any existing global cluster, a new global cluster is created, and the target vector is used as the initial centroid of the global cluster. Then, a clustering algorithm is used to re-cluster all vectors containing the target vector to determine the new global cluster partition. The global index layer, regional index layer, and local index layer are reconstructed in sequence. The new global cluster is added to the global index layer, the cluster type is updated, and all vectors are redistributed to the new global cluster. The centroid of the global index and the regional index pointer are updated. For example, the ancient book restoration vector is used as the centroid of G101, and some vectors in the original historical document cluster are migrated to G101.

[0047] For example, the vector library initially contains two global clusters: pop music and rock music. A new Chinese pop song A (influence 0.2) is added, and a local incremental update is performed to insert it into the pop music local cluster. A new vaporwave electronic music B (influence 0.6) is added, and a regional reconstruction is performed to create an experimental electronic sub-cluster under the electronic music regional cluster. A new chanting music C (influence 0.9) is added, and a full reconstruction is performed to create a new chanting music global cluster and redefine all music types.

[0048] Furthermore, by selecting the appropriate index layer to update based on the scope of the update operation, the problem of triggering a global reconstruction due to local changes in traditional methods can be avoided. At the same time, the incremental synchronization mechanism between each layer maintains consistency and ensures the accuracy of query results.

[0049] In this embodiment of the invention, a hierarchical update mechanism can be used to flexibly respond to dynamic changes in the vector library while maintaining query efficiency, thereby achieving adaptive index maintenance that is locally fine-tuned and globally controllable.

[0050] In this embodiment of the invention, updating the multi-level index structure based on the index data corresponding to the updated target vector data in the vector library includes: When the updated target vector data in the vector library is in the local index layer, the edge attribute between the target vector data and the vector data in the local index layer is constructed based on the index data corresponding to the updated target vector data in the vector library. Update the nearest neighbor graph corresponding to the local index layer according to the edge attributes, and update the local index layer in the multi-level index structure according to the updated nearest neighbor graph; When the updated target vector data in the vector library is in the region index layer, the cluster center coordinates corresponding to the region cluster are updated, the region cluster is updated according to the cluster center coordinates, and the region index layer in the multi-level index structure is updated according to the updated region cluster. When the updated target vector data in the vector library is in the global index layer, the vector library is spatially remapped based on the updated target vector data in the vector library to obtain the target global cluster center; Update the global index layer in the multi-level index structure according to the target global cluster center.

[0051] In detail, within the local indexing layer, connections are established between the target vector and existing vectors. Specifically, for each vector in the local indexing layer, cosine similarity or Euclidean distance is calculated, and the similarity is converted into edge weights. The target vector is then added as a new node to the nearest neighbor graph, and additional nodes are added based on the edge weights. The strongest connection, For hyperparameters, such as Then, for the existing nodes connected to the new node, check whether their nearest neighbor list needs to be updated, and update the metadata of the local index layer. The changes are synchronized to the regional index layer through pointer association. When the vectors in the regional cluster change, the new centroid of the regional cluster or the mean of all vectors in the cluster is recalculated. Then, according to the position of the new vector, the boundary of the regional cluster is adjusted, and the vector density in the cluster is calculated. If it is lower than the threshold, splitting or merging is triggered. The regional cluster is divided into secondary parts using hierarchical clustering or DBSCAN, and the sub-cluster pointers are updated. For example, after adding vaporwave electronic vectors, the centroid of the electronic music regional cluster shifts towards the experimental electronic direction, and the boundary expands to include the new type. Then, the cluster center, boundary and sub-cluster pointers of the regional index layer are updated, and the changes are synchronized to the global index layer.

[0052] Specifically, when the addition of a vector causes a significant change in the global cluster structure (such as the creation of a new category), global features are extracted from the updated target vector data in the vector library, the vector space is re-partitioned, a new global cluster is generated, and the original vectors are redistributed to the new global cluster. For each new global cluster, its centroid is calculated as the new cluster center, and then the cluster center and boundary of the global index layer are reconstructed, the mapping relationship from the global cluster to the regional cluster is updated, and all regional index layers are notified to update their parent cluster information.

[0053] Furthermore, the regional index layer, as an intermediate layer, is responsible for coordinating local and global changes, receiving update notifications from the local index layer, updating cluster statistics, and reporting major changes, such as cluster splits and mergers, to the global index layer. The global index layer maintains the highest-level structure, controls the update scope, determines whether spatial remapping needs to be triggered, and synchronizes global structural changes to all regional index layers.

[0054] Furthermore, compared with the traditional full reconstruction method, the incremental update algorithm of this invention has achieved a significant improvement in computational efficiency. Through a selective update strategy, the algorithm can reduce the time complexity of the update operation from O(n²) to O(k·log n), where k is the number of vectors within the scope of influence, which is much smaller than the total number of vectors n. When processing millions of vector data, the algorithm of this invention can reduce the computation time by more than 80% compared with the traditional method, while maintaining a query accuracy of more than 95%. This performance improvement is particularly important when processing real-time data streams and can meet the application requirements of high-frequency updates.

[0055] As can be seen, in the above scheme, the optimal update strategy is adaptively selected through a multi-level index structure, which includes a global index layer, a regional index layer, and a local index layer. The global index layer maintains coarse-grained information about the overall data distribution, the regional index layer manages medium-grained data clustering information, and the local index layer processes fine-grained vector proximity relationships to support efficient incremental updates. Based on the changing characteristics of the vector data and the system load, the optimal update strategy is adaptively selected, which significantly reduces the frequency of full reconstruction while ensuring update accuracy, thereby solving the technical problem of low efficiency in adaptive updates of vector data.

[0056] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] In one embodiment, an adaptive update apparatus for vector data is provided, which corresponds one-to-one with the adaptive update method for vector data in the above embodiments. For example... Figure 5 As shown, the adaptive update device for this vector data includes an influence region identification module 101, an adaptive update strategy determination module 102, and a multi-level index structure update module 103. Detailed descriptions of each functional module are as follows: The influence region identification module 101 is used to extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence region of the feature fingerprint in the vector library through the multi-level index structure of the vector library; The adaptive update strategy determination module 102 is used to determine the adaptive update strategy of the target vector data in the vector library based on the affected area and the preset update triggering event of the vector library; The multi-level index structure update module 103 is used to update the target vector data to the vector library through the adaptive update strategy, and update the multi-level index structure based on the index data corresponding to the updated target vector data in the vector library.

[0058] In one embodiment, the influence region identification module 101, when performing the extraction of feature fingerprints of target vector data stored in a preset vector library, is used to: Convert the data type of the target vector data stored in the preset vector library; Calculate the hash value corresponding to the target vector data after data type conversion; Identify the basic metadata and business metadata corresponding to the target vector data; The basic metadata, the business metadata, and the hash value are encapsulated into the feature fingerprint of the target vector data.

[0059] In one embodiment, the influence region identification module 101, before performing the step of identifying the influence region of the feature fingerprint within the vector library through the multi-level index structure of the vector library, is configured to: Perform cluster analysis on all vectors in the vector library to obtain global clusters, and determine the global index layer corresponding to the vector library based on the global clusters. The global cluster is divided into regional clusters according to the attribute type corresponding to the global cluster, and the regional index layer corresponding to the vector library is determined according to the regional clusters. Extract the vector data and vector relationships corresponding to the region clusters, and construct an index file for the region clusters based on the vector data and vector relationships; The region clusters are divided into local clusters according to the index file, and the local index layer corresponding to the vector library is determined based on the local clusters. The vector library is constructed using the global index layer, the regional index layer, and the local index layer, forming a multi-level index structure.

[0060] In one embodiment, the influence region identification module 101, when performing the identification of the influence region of the feature fingerprint within the vector library through the multi-level index structure of the vector library, is configured to: The feature fingerprint is matched with the cluster type corresponding to the global cluster in the global index layer to obtain the global candidate cluster corresponding to the feature fingerprint; Calculate the global similarity between the target vector data corresponding to the feature fingerprint and the cluster centroid of the global candidate cluster, and select the global candidate cluster with the largest global similarity as the target global cluster; Identify the region cluster corresponding to the global cluster based on the region index pointer corresponding to the target global cluster; Calculate the boundary distance between the target vector data and the region cluster, and filter the target region clusters in the region clusters based on the boundary distance; Identify the local index layer corresponding to the target vector data based on the index file corresponding to the target region cluster, and calculate the local similarity between the target vector data and each vector data in the local index layer; When the local similarity is greater than a preset similarity threshold, the vector data in the local index layer is used as the target vector data, and the influence area of ​​the feature fingerprint in the vector library is determined according to the index range of the target vector data.

[0061] In one embodiment, the influence region identification module 101, when performing the filtering of target region clusters in the region clusters based on the boundary distance, is configured to: When the boundary distance is less than a preset distance threshold, the region cluster is designated as the first region cluster. Identify whether the density of vector data in the first region cluster exceeds a preset density threshold; When the density of vector data in the first region cluster exceeds the density threshold, the first region cluster is taken as the target region cluster.

[0062] When the density of vector data in the first region cluster does not exceed the density threshold, the parent region cluster corresponding to the first region cluster is identified. The target region cluster within the region cluster is determined based on the parent region cluster.

[0063] In one embodiment, the adaptive update strategy determination module 102, when performing the execution of determining the adaptive update strategy of the target vector data in the vector library based on the affected area and the preset update trigger event of the vector library, is used to: Calculate the target distance between the target vector data and the cluster centers of the affected region, and determine the influence degree of the affected region based on the target distance; The update strategy for the target vector data is determined based on the influence degree and the preset influence threshold. The adaptive update strategy for the target vector data in the vector library is determined by the preset update trigger event and the update strategy.

[0064] In one embodiment, the multi-level index structure update module 103, when updating the target vector data to the vector library using the adaptive update strategy, is configured to: When the adaptive update strategy is a local incremental update strategy, the vector data in the influence range is fused to obtain fused vector data; The fused vector data is incrementally inserted into the local index layer corresponding to the vector library; When the adaptive update strategy is a regional reconstruction strategy, the region cluster corresponding to the target vector data is identified, the region cluster is reconstructed according to the target vector data, and the regional index position of the target vector data in the vector library is determined according to the reconstructed region cluster. Update the target vector data to the corresponding region index layer of the vector library according to the region index position; When the adaptive update strategy is a full reconstruction strategy, the target global cluster corresponding to the target vector data is identified, the target vector data is updated in the target global cluster according to the preset initial position, and the global cluster corresponding to the vector library is reconstructed according to the updated target global cluster. The global index position of the target vector data in the vector library is determined based on the reconstructed global cluster, and the target vector data is updated to the global index layer corresponding to the vector library according to the global index position.

[0065] This invention provides an adaptive update device for vector data. It adaptively selects the optimal update strategy through a multi-level index structure, which includes a global index layer, a regional index layer, and a local index layer. The global index layer maintains coarse-grained information about the overall data distribution, the regional index layer manages medium-grained data clustering information, and the local index layer processes fine-grained vector proximity relationships to support efficient incremental updates. Based on the changing characteristics of the vector data and the system load, the optimal update strategy is adaptively selected, significantly reducing the frequency of full reconstruction while ensuring update accuracy, thereby solving the technical problem of low efficiency in adaptive updates of vector data.

[0066] Specific limitations regarding the adaptive update device for vector data can be found in the limitations of the adaptive update method for vector data described above, and will not be repeated here. Each module in the aforementioned adaptive update device for vector data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0067] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements a server-side function or step of an adaptive update method for vector data.

[0068] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an adaptive update method for vector data.

[0069] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence area of ​​the feature fingerprint in the vector library through the multi-level index structure of the vector library; The adaptive update strategy for the target vector data in the vector library is determined based on the affected area and the preset update triggering events of the vector library. The target vector data is updated to the vector library using the adaptive update strategy, and the multi-level index structure is updated based on the index data corresponding to the updated target vector data in the vector library.

[0070] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence area of ​​the feature fingerprint in the vector library through the multi-level index structure of the vector library; The adaptive update strategy for the target vector data in the vector library is determined based on the affected area and the preset update triggering events of the vector library. The target vector data is updated to the vector library using the adaptive update strategy, and the multi-level index structure is updated based on the index data corresponding to the updated target vector data in the vector library.

[0071] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0072] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0073] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0074] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

[0075] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An adaptive update method for vector data, characterized in that, include: Extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence area of ​​the feature fingerprint in the vector library through the multi-level index structure of the vector library; The adaptive update strategy for the target vector data in the vector library is determined based on the affected area and the preset update triggering events of the vector library. The target vector data is updated to the vector library using the adaptive update strategy, and the multi-level index structure is updated based on the index data corresponding to the updated target vector data in the vector library.

2. The adaptive update method for vector data as described in claim 1, characterized in that, The step of extracting the feature fingerprint of the target vector data stored in the preset vector library includes: Convert the data type of the target vector data stored in the preset vector library; Calculate the hash value corresponding to the target vector data after data type conversion; Identify the basic metadata and business metadata corresponding to the target vector data; The basic metadata, the business metadata, and the hash value are encapsulated into the feature fingerprint of the target vector data.

3. The adaptive update method for vector data as described in claim 1, characterized in that, Before identifying the influence region of the feature fingerprint within the vector library through the multi-level index structure of the vector library, the method further includes: Perform cluster analysis on all vectors in the vector library to obtain global clusters, and determine the global index layer corresponding to the vector library based on the global clusters. The global cluster is divided into regional clusters according to the attribute type corresponding to the global cluster, and the regional index layer corresponding to the vector library is determined according to the regional clusters. Extract the vector data and vector relationships corresponding to the region clusters, and construct an index file for the region clusters based on the vector data and vector relationships; The region clusters are divided into local clusters according to the index file, and the local index layer corresponding to the vector library is determined based on the local clusters. The vector library is constructed using the global index layer, the regional index layer, and the local index layer, forming a multi-level index structure.

4. The adaptive update method for vector data as described in claim 3, characterized in that, The step of identifying the influence region of the feature fingerprint within the vector library through the multi-level index structure of the vector library includes: The feature fingerprint is matched with the cluster type corresponding to the global cluster in the global index layer to obtain the global candidate cluster corresponding to the feature fingerprint; Calculate the global similarity between the target vector data corresponding to the feature fingerprint and the cluster centroid of the global candidate cluster, and select the global candidate cluster with the largest global similarity as the target global cluster; Identify the region cluster corresponding to the global cluster based on the region index pointer corresponding to the target global cluster; Calculate the boundary distance between the target vector data and the region cluster, and filter the target region clusters in the region clusters based on the boundary distance; Identify the local index layer corresponding to the target vector data based on the index file corresponding to the target region cluster, and calculate the local similarity between the target vector data and each vector data in the local index layer; When the local similarity is greater than a preset similarity threshold, the vector data in the local index layer is used as the target vector data, and the influence area of ​​the feature fingerprint in the vector library is determined according to the index range of the target vector data.

5. The adaptive update method for vector data as described in claim 4, characterized in that, The step of filtering the target region clusters in the region clusters based on the boundary distance includes: When the boundary distance is less than a preset distance threshold, the region cluster is designated as the first region cluster. Identify whether the density of vector data in the first region cluster exceeds a preset density threshold; When the density of vector data in the first region cluster exceeds the density threshold, the first region cluster is taken as the target region cluster. When the density of vector data in the first region cluster does not exceed the density threshold, the parent region cluster corresponding to the first region cluster is identified. The target region cluster within the region cluster is determined based on the parent region cluster.

6. The adaptive update method for vector data as described in claim 1, characterized in that, The step of determining the adaptive update strategy for the target vector data in the vector library based on the affected area and the preset update trigger events of the vector library includes: Calculate the target distance between the target vector data and the cluster centers of the affected region, and determine the influence degree of the affected region based on the target distance; The update strategy for the target vector data is determined based on the influence degree and the preset influence threshold. The adaptive update strategy for the target vector data in the vector library is determined by the preset update trigger event and the update strategy.

7. The adaptive update method for vector data as described in claim 1, characterized in that, The step of updating the target vector data to the vector library using the adaptive update strategy includes: When the adaptive update strategy is a local incremental update strategy, the vector data in the influence range is fused to obtain fused vector data; The fused vector data is incrementally inserted into the local index layer corresponding to the vector library; When the adaptive update strategy is a regional reconstruction strategy, the region cluster corresponding to the target vector data is identified, the region cluster is reconstructed according to the target vector data, and the regional index position of the target vector data in the vector library is determined according to the reconstructed region cluster. Update the target vector data to the corresponding region index layer of the vector library according to the region index position; When the adaptive update strategy is a full reconstruction strategy, the target global cluster corresponding to the target vector data is identified, the target vector data is updated in the target global cluster according to the preset initial position, and the global cluster corresponding to the vector library is reconstructed according to the updated target global cluster. The global index position of the target vector data in the vector library is determined based on the reconstructed global cluster, and the target vector data is updated to the corresponding global index layer of the vector library according to the global index position.

8. An adaptive update device for vector data, characterized in that, include: The influence region identification module is used to extract the feature fingerprint of the target vector data stored in the preset vector library, and identify the influence region of the feature fingerprint in the vector library through the multi-level index structure of the vector library; An adaptive update strategy determination module is used to determine the adaptive update strategy of the target vector data in the vector library based on the affected area and the preset update triggering event of the vector library; The multi-level index structure update module is used to update the target vector data to the vector library through the adaptive update strategy, and update the multi-level index structure based on the index data corresponding to the updated target vector data in the vector library.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the adaptive update method for vector data as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the adaptive update method for vector data as described in any one of claims 1 to 7.

Images