A method and system for incremental retrieval and index updating of heterogeneous data

Abstract

This invention provides a method and system for incremental retrieval index update of heterogeneous data, relating to the field of data processing technology. The method includes the following steps: batch transferring newly added information representation values ​​from the real-time cache layer to the recently active layer and performing local index optimization; analyzing the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer and establishing jump edges; identifying invalid pointers and duplicate information fragments in the index for quickly finding similar information, and deleting invalid pointers and merging duplicate information fragments. This invention aims to solve the problems of traditional incremental update methods failing to effectively integrate different types of data when intelligent question-answering systems process massive, heterogeneous data, leading to limited retrieval efficiency and accuracy, as well as inconsistencies in internal index logic, invalid pointers, and duplicate information fragments, effectively improving system response speed and query result confidence.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and system for incremental retrieval and index updating of heterogeneous data. Background Technology

[0002] In today's rapidly developing information technology landscape, intelligent question-answering systems have become crucial tools for improving information retrieval efficiency. However, ensuring rapid information retrieval and accurate matching is a significant challenge, especially when dealing with massive amounts of diverse data from various sources. Traditional processing methods often struggle to effectively integrate different data types, leading to bottlenecks in meaning comprehension and retrieval efficiency. For instance, when a system needs to integrate large amounts of historical archives, the age, inconsistent formats, and varying quality of these materials pose a severe test to the core functions of existing intelligent question-answering systems. These challenges extend beyond the complexity of the data itself, penetrating into every aspect of data processing, information organization, and retrieval mechanisms. This makes it difficult for the system to consistently provide high-confidence query results while maintaining response speed, even impacting the user's actual experience.

[0003] Specifically, during the continuous integration and updating of the fast similar information search index structure in the intelligent question-answering system, existing incremental update methods fail to effectively handle this heterogeneity when merging index files due to significant differences in the distribution of numerical representations of new and old data, as well as inconsistencies in the meaning definitions within the information repository. The construction of the fast similar information search index is highly dependent on the spatial distribution characteristics of the numerical representations of the information. When the distribution of newly added numerical representations differs significantly from that in the original index, the method may produce internal logical inconsistencies when attempting to integrate new nodes into the existing structure. For example, when constructing the index graph, if the neighbor selection strategy for new nodes fails to fully consider their connectivity with heterogeneous data, invalid pointers to logically deleted or merged old information fragments may not be updated or cleaned up in a timely manner. Simultaneously, improper handling of heterogeneous data may also lead to duplicate information fragments within the index, meaning that identical or highly similar data is repeatedly stored or referenced in the index, increasing the complexity of the index and the traversal path during queries. It is important to emphasize that these problems are not due to damage to the physical storage medium, but rather defects in the internal logical structure of the index; they occur intermittently in the index file and are not universally present.

[0004] When a retrieval request happens to encounter information fragments with internal logical errors, the system will trigger internal validation failures or data inconsistency anomalies during the loading and parsing of index data. For example, if a query for information represents a value and encounters an invalid pointer while traversing the index graph, the system will be unable to find the expected target node, or when accessing a duplicate information fragment, it will find that its internal data does not match expectations. To maintain data integrity and prevent system crashes, the intelligent question-answering system automatically triggers an error handling mechanism. When this mechanism detects an internal logical error in the index, it will abandon the use of that part of the index and fall back to a more time-consuming step-by-step comparison search method for secondary verification. This fallback operation significantly increases the average waiting time for some queries, especially when historical data is involved or different forms of related queries are required, and the originally promised second-level response capability cannot be guaranteed.

[0005] There is currently no effective technical solution to the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for incremental retrieval index update of heterogeneous data, which aims to solve the problems that traditional incremental update methods cannot effectively integrate different types of data when intelligent question answering systems process massive and heterogeneous data, resulting in limited retrieval efficiency and accuracy, as well as inconsistencies in the internal logic of the index, invalid pointers and duplicate information fragments, and effectively improve the system response speed and the confidence of query results.

[0007] In a first aspect, the present invention provides a method for updating an incremental retrieval index for heterogeneous data, comprising the following steps: Construct a multi-level index for fast similar information lookup; the fast similar information lookup index includes a historical stable layer, a recently active layer, and a real-time cache layer; The system receives newly added information representation values ​​and transfers them in batches from the real-time cache layer to the recently active layer. In the recently active layer, the system performs local index optimization on the newly added information representation values ​​to achieve incremental updates of the newly added information representation values. Before merging the information representation values ​​in the recently active layer into the historical stable layer, the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer are analyzed to identify the correlation between the two information representation values, and a jump edge connecting the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer is established in the fast search similar information index. Identify invalid pointers and duplicate information fragments in the fast lookup similar information index, delete invalid pointers and merge duplicate information fragments to achieve topological self-healing operation of the fast lookup similar information index.

[0008] The heterogeneous data incremental retrieval index update method provided by this invention constructs a multi-level index structure, realizes hierarchical management and incremental updates of heterogeneous data, and effectively solves the problems of heterogeneous data integration, index efficiency and accuracy through skip edges and topology self-healing mechanism. It avoids the inconsistency of index logic and the decline in retrieval performance caused by data heterogeneity in traditional methods, thereby improving the overall performance and user experience of intelligent question answering system.

[0009] Secondly, the present invention provides a heterogeneous data incremental retrieval index update system, comprising: The building module is used to construct a multi-level index for fast similar information lookup; the fast similar information lookup index includes a historical stable layer, a recently active layer, and a real-time cache layer; The update module is used to receive newly added information representation values, and to transfer the newly added information representation values ​​from the real-time cache layer to the recently active layer in batches. In the recently active layer, the local index optimization is performed on the newly added information representation values ​​to achieve incremental updates of the newly added information representation values. The association module is used to analyze the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer before merging the information representation values ​​in the recent active layer into the historical stable layer, so as to identify the relationship between the information representation values ​​of the two, and to establish a jump edge connecting the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer in the quick search similar information index. The sorting module is used to identify invalid pointers and duplicate information fragments in the quick lookup similar information index, and delete invalid pointers and merge duplicate information fragments to achieve topological self-healing operation of the quick lookup similar information index.

[0010] As can be seen from the above, the heterogeneous data incremental retrieval index update method provided by this invention achieves hierarchical management and efficient storage of heterogeneous data by constructing a multi-level index for fast similar information retrieval, including a historical stable layer, a recently active layer, and a real-time cache layer. This method can receive newly added information representation values ​​and transfer them in batches from the real-time cache layer to the recently active layer, and perform local index optimization in the recently active layer, thereby achieving incremental updates of newly added information representation values. This effectively solves the problem of low efficiency in traditional methods when dealing with differences in the distribution of old and new data. Furthermore, before merging the information representation values ​​in the recently active layer into the historical stable layer, this application analyzes the two to identify correlations and establishes skip edges, enabling the retrieval process to skip intermediate index levels and achieve direct logical addressing across levels, significantly improving retrieval efficiency. More importantly, this method can identify and delete invalid pointers and merge duplicate information fragments, realizing a topological self-healing operation for quickly finding similar information indexes. It effectively solves the problems of inconsistency in the internal logic of the index, invalid pointers, and duplicate information fragments caused by improper handling of heterogeneous data in the prior art. It avoids the system falling back to the time-consuming one-by-one comparison search method, thereby ensuring the intelligent question answering system's second-level response capability and high-confidence query results when processing massive heterogeneous data, and overcoming the shortcomings of limited retrieval efficiency and accuracy in the prior art.

[0011] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0012] Figure 1 This is a flowchart of a heterogeneous data incremental retrieval index update method provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of a heterogeneous data incremental retrieval index update system provided in an embodiment of the present invention.

[0014] Label Explanation: 100. Build module; 200. Update module; 300. Association module; 400. Organize module; 500. Monitor module; 600. Analyze module. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0016] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0017] During the data retrieval index update process of an intelligent question-answering system, when newly added heterogeneous data differs from historical data in the distribution of information representation values, the incremental update operation of quickly finding similar information in the index is prone to index logic errors. Specifically, the distribution difference manifests as a low overlap between the distribution of the information representation values ​​of the new data in the feature space and that of the historical data, leading to invalid pointers and duplicate information fragments during the index merging phase. Furthermore, invalid pointers refer to references in the index structure that point to data nodes that have been logically deleted or merged, while duplicate information fragments refer to the same or highly similar data being stored multiple times in the index. Consequently, when the retrieval process encounters these erroneous fragments, it triggers the system error handling mechanism, forcing the system to revert to a one-to-one comparison search method, resulting in increased query response time and higher system resource consumption.

[0018] For example, in a scenario where a trade union's intelligent question-and-answer system integrates historical archives of "model workers' deeds," these archives contain a large number of non-standard format scanned documents and outdated video materials. Specifically, the scanned documents originate from scanning equipment from different eras, resulting in inconsistent resolutions and handwritten annotations, leading to errors in the numerical representation of text after recognition. The video materials use outdated encoding formats, resulting in noise in the numerical representation of the transcribed information. During index updates, when the recently active layer receives this new data, the significant difference in distribution between it and the historical stable layer data causes the merging operation to fail to properly handle correlations, resulting in invalid pointers to deleted nodes and duplicate content fragments. When the system retrieves queries related to historical deeds, the presence of invalid pointers prevents direct addressing, triggering a fallback mechanism, prolonging user waiting time, and forcibly adding a confirmation step to the user interface, thus reducing interaction efficiency.

[0019] If the aforementioned indexing logic errors are not addressed, the internal inconsistencies in the index structure of the intelligent question-answering system will gradually accumulate during the continuous integration of heterogeneous data. Specifically, the increase in invalid pointers and duplicate fragments will lead to frequent triggering of error handling mechanisms, making it impossible for the system to maintain approximate search capabilities. Furthermore, the continuous increase in retrieval response time will affect user trust in the system, while the forced addition of interaction steps further weakens the user experience. Consequently, the system's reliability in critical business scenarios is threatened, potentially failing to meet real-time query requirements and ultimately causing partial system failure.

[0020] This invention provides a method for updating an incremental retrieval index for heterogeneous data, comprising the following steps: Construct a multi-level index for fast similar information lookup; the fast similar information lookup index includes a historical stable layer, a recently active layer, and a real-time cache layer; the historical stable layer is used to store verified old data indexes; the recently active layer is used to store newly added data indexes; the real-time cache layer is used to store data indexes generated in real time; The system receives newly added information representation values ​​and transfers them in batches from the real-time cache layer to the recently active layer. In the recently active layer, the system performs local index optimization on the newly added information representation values ​​to achieve incremental updates of the newly added information representation values. Before merging the information representation values ​​in the recently active layer into the historical stable layer, the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer are analyzed to identify the correlation between the two information representation values. In the fast lookup similar information index, a jump edge is established to connect the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer. The jump edge connection is used to skip the intermediate index levels in the fast lookup similar information index during the retrieval process, realizing direct logical addressing across levels. Identify invalid pointers and duplicate information fragments in the fast lookup similar information index, delete invalid pointers and merge duplicate information fragments to achieve topological self-healing operation of the fast lookup similar information index.

[0021] For ease of understanding, the following explains some key terms in this embodiment: Fast Similar Information Search Index: This index employs a multi-level structure designed for efficient retrieval of similar information within massive amounts of heterogeneous data. Its core lies in optimizing retrieval efficiency and data update flexibility through hierarchical storage and management of data indexes in different states. The index is typically constructed using approximate nearest neighbor search algorithms such as HNSW (Hierarchical Navigable Small World) to achieve fast searching within a multi-dimensional vector space.

[0022] Historical Stable Layer: This layer is a component of the index for quickly finding similar information and is specifically designed to store well-validated and calibrated older data indexes. The data index structure in this layer is stable, primarily used for quickly finding confirmed, older information. It is characterized by low data change frequency and high requirements for query accuracy and stability.

[0023] Recently Active Layer: This layer is another component of the fast-search similar information index, used to store recently added but not yet fully processed new data. The data index structure of this layer is relatively flexible, allowing for frequent partial updates and optimizations to accommodate the rapid influx of new data and the need for initial processing.

[0024] Real-time caching layer: This layer is the top layer in the index for quickly finding similar information, serving as the fastest and smallest temporary storage area. Its main function is to process extremely small batches of newly generated data that need to be immediately recognized and responded to by the system, ensuring that new information is received and initially processed by the system at millisecond speeds.

[0025] Numerical representation of information: This numerical value is a vectorized representation of the original heterogeneous data (such as text, images, videos, etc.) after unified representation learning. These values ​​carry the semantic information and features of the original data in a multi-dimensional vector space, and are the foundation for similarity calculation and index construction.

[0026] Skip edges: These are special connections in the fast lookup similarity index, used to directly connect new information groups in the recently active layer with old information groups in the historically stable layer. Skip edges have weights representing the strength of semantic association. Their function is to skip intermediate index levels in the fast lookup similarity index during the retrieval process, achieving direct logical addressing across levels, thereby improving retrieval efficiency.

[0027] Topology self-healing operation: This operation is a maintenance procedure designed to comprehensively inspect and repair the index structure for quickly finding similar information. By identifying and removing invalid pointers in the index, and identifying and merging duplicate information fragments in the index, this operation ensures the logical consistency of the index structure and retrieval performance.

[0028] This application proposes a method for updating indexes for incremental retrieval of heterogeneous data, aiming to solve the problems of index logic errors and decreased retrieval efficiency caused by the difference in distribution between new and old data during the index update process for incremental retrieval of heterogeneous data.

[0029] This method first constructs a multi-level index for fast lookup of similar information. This index includes a historically stable layer, a recently active layer, and a real-time cache layer. The historically stable layer stores verified older data indexes; its structure is stable and primarily used for fast lookup of confirmed older information. The recently active layer stores newly added data indexes; its structure is more flexible, allowing for frequent local updates and optimizations. The real-time cache layer stores data indexes generated in real-time, serving as the fastest and smallest temporary storage area, ensuring that new information is received and initially processed by the system within milliseconds. For example, the historically stable layer and the recently active layer can be constructed using an HNSW index, while the real-time cache layer can use simple in-memory data structures, such as fast lookup tables or small vector indexes.

[0030] Next, the method receives newly added information representation values ​​and transfers them in batches from the real-time cache layer to the recently active layer. In the recently active layer, local index optimization is performed on the newly added information representation values ​​to achieve incremental updates. For example, when the system receives new information representation values, they can first be placed in the real-time cache layer, processed initially, and then transferred in batches to the recently active layer. In the recently active layer, the proximity relationships of these newly added information representation values ​​can be recalculated, and their connections in the index graph can be adjusted, thereby achieving local index optimization.

[0031] Furthermore, before merging the information representation values ​​from the recently active layer into the historically stable layer, this method analyzes the information representation values ​​in the recently active layer and the historically stable layer to identify the correlation between their information representation values. When such a deep semantic correlation is found, a jump edge is established in the fast lookup similar information index to connect the information representation values ​​in the recently active layer and the historically stable layer. The jump edge connection is used to skip intermediate index levels in the fast lookup similar information index during the retrieval process, achieving direct logical addressing across levels. For example, the system can initiate a "meaning reconstruction analysis" program to compare the information representation values ​​of new and old data, identifying information groups that appear significantly different on the surface but are closely related in terms of deep meaning. When such a deep semantic correlation is found, the system adds a special jump edge to the index graph, directly connecting the new information group in the recently active layer and the old information group in the historically stable layer. The jump edge can have a weight representing the strength of the semantic correlation; the weight value can be set based on the semantic similarity score.

[0032] Finally, this method identifies invalid pointers and duplicate information fragments in the fast lookup similarity information index, and deletes invalid pointers and merges duplicate information fragments to achieve topological self-healing of the fast lookup similarity information index. For example, the system can periodically start a "topology self-healing" maintenance program to comprehensively check and repair the fast lookup similarity information index structure. This program can identify and delete invalid pointers in the index, such as connecting lines pointing to information representation values ​​that have been deleted or merged. Simultaneously, the program can identify and merge duplicate information fragments in the index, such as multiple index paths ultimately pointing to the same document, or information representation values ​​that are semantically almost identical but exist as independent nodes. The identification of duplicate information fragments can be achieved by calculating the similarity of information representation values ​​within a local neighborhood.

[0033] The core technical concept of this solution lies in treating the index for quickly finding similar information as an organism with a lifecycle. By dividing the index into a historically stable layer, a recently active layer, and a real-time cache layer, it differentiates information of different "ages" and "stability." Based on this, by deeply mining the semantic relationships between old and new data, even if their information representation numerical distributions differ significantly, it can intelligently establish cross-level jump edges to bridge semantic inconsistencies. Simultaneously, a topology self-healing operation is employed to periodically self-check and repair the index structure, cleaning up invalid connections and redundant information to ensure the logical integrity of the index and search efficiency. This combination of hierarchical management, meaning-driven connections, and self-repair enables the system to proactively adapt to the challenges of massive heterogeneous data, fundamentally avoiding index logic defects and guaranteeing continuous and efficient searching.

[0034] The following example will provide a more detailed explanation of the above technical solution: Suppose an intelligent question-answering system needs to process a large amount of historical union archives data, including text, images, and videos, in diverse formats and of varying quality. Traditional indexing and updating methods are prone to indexing logic errors and decreased retrieval efficiency when dealing with such heterogeneous data, due to the significant differences in the numerical distribution of information representation between the old and new data.

[0035] To address this issue, this application proposes a method for incremental retrieval and index updating of heterogeneous data. First, the system constructs a multi-level index for fast searching of similar information. This index includes a historically stable layer, a recently active layer, and a real-time cache layer. The historically stable layer stores verified old union bylaws, historical activity records, and other data indexes; these data are stable and frequently queried. The recently active layer stores newly added data indexes such as the deeds of model workers and employee innovation achievements; this data may still be being updated and improved. The real-time cache layer stores real-time question-and-answer requests submitted by employees or newly generated temporary data indexes.

[0036] When new data on exemplary deeds (e.g., a new scanned document or video clip) is digitized and generated into numerical representations, these values ​​are first received and temporarily stored in a real-time cache layer. Subsequently, the system periodically transfers these newly added numerical representations from the real-time cache layer to the recently active layer. In the recently active layer, the system performs local index optimization on this new data. For example, the system recalculates the proximity relationships of this new data within the recently active layer and adjusts their connections in the index graph to ensure that the new data can be effectively integrated into the index structure of the recently active layer, achieving incremental updates.

[0037] Before merging the locally optimized data on model worker achievements from the recently active layer into the historical stable layer, the system initiates a "meaning reconstruction analysis" procedure. This procedure analyzes the correlation between new data in the recently active layer and old data in the historical stable layer (e.g., existing model worker selection criteria, honorary level information). Even if the new and old data differ in the numerical distribution of information representation, if the semantic understanding module determines that they are highly correlated in meaning (e.g., a certain achievement described by the new data highly matches a certain selection criterion in the historical data), the system will identify this deep semantic correlation. Once the correlation is identified, the system establishes jump edges connecting the new data in the recently active layer and the old data in the historical stable layer in a quick similarity information search index. For example, a jump edge can directly connect a newly added model worker achievement with the corresponding "innovation achievement" selection criterion in the historical stable layer. This jump edge has a weight representing the strength of the semantic correlation, for example, set according to semantic similarity scores. In subsequent searches, when users query the deeds of model workers related to "innovative achievements", the system can jump directly from the recently active layer to the historical stable layer through this jump edge, skipping the intermediate index levels and realizing direct logical addressing across levels, thereby significantly improving search efficiency.

[0038] In addition, the system periodically initiates a "topology self-healing" maintenance procedure. This procedure comprehensively checks and repairs the index structure for quickly finding similar information. For example, it identifies and removes potentially invalid pointers in the index, such as connections pointing to old temporary data that has been deleted or merged. Simultaneously, it identifies and merges duplicate information fragments in the index. For instance, if multiple index paths ultimately point to the same document on the deeds of a model worker, or if information representations are semantically almost identical but exist as independent nodes, the system merges them to reduce redundancy and optimize the index structure. Duplicate information fragments can be identified by calculating the similarity of information representation values ​​within local neighborhoods (e.g., a cosine similarity higher than 0.99).

[0039] Using the above method, when user A queries the specific deeds of a historical model worker in the intelligent question-answering system, even if the deeds are newly added heterogeneous data, the system can quickly and accurately locate the relevant information through local optimization of the recently active layer and the skip edge established with the historical stable layer. The existence of the skip edge allows the system to skip intermediate layers and directly access semantically related old data, avoiding the problem of invalid retrieval paths caused by differences in data distribution in traditional methods. At the same time, the topology self-healing operation ensures the logical consistency of the index structure, avoiding the impact of invalid pointers and duplicate information fragments on retrieval performance. Compared with traditional methods that may fall back to time-consuming one-by-one comparison search when encountering heterogeneous data, the method of this application can continuously provide high-efficiency and high-accuracy retrieval services, significantly improving the user experience and solving the problems of decreased retrieval response speed and inability to guarantee the accuracy of query results in traditional methods.

[0040] In some embodiments, the steps of receiving newly added information representation values ​​and transferring the newly added information representation values ​​from the real-time cache layer to the recently active layer in batches include: The newly added information represents the generation time and access frequency of the value; When the generation time is less than or equal to the preset first time threshold, the newly added information representation value will be allocated to the real-time cache layer; When the generation time is greater than the first time threshold and the access frequency is greater than the preset frequency threshold, the newly added information will be assigned to the recently active layer.

[0041] The process involves several key metrics: First, the generation time and access frequency of newly added information representation values. Generation time refers to the timestamp when the information representation value is first created or received by the system, measuring its "freshness" or "timeliness." Possible implementations include: automatically recording the timestamp during data ingestion; or, if the data originates from an external system, parsing the creation time metadata inherent in the data source. Access frequency refers to the number of times the information representation value is retrieved or accessed within a specific time window, assessing its "activity" or "importance." Possible implementations include: recording and counting each access event through system logs; or, estimating access frequency using indirect metrics such as cache hit rate and user interaction behavior. When the generation time is less than or equal to a preset first time threshold, the newly added information representation value is allocated to the real-time cache layer. The first time threshold is a pre-defined time length used to define the freshness of the data, providing a temporal standard for data allocation in the real-time cache layer. Possible implementation methods include: setting the first time threshold to, for example, 1 minute, 5 minutes, or 10 minutes, with the specific value adjusted according to different real-time requirements of the system; or, the threshold can be dynamically adjusted based on system load, data inflow rate, etc. Assigning to the real-time cache layer refers to logically or physically placing the information representation values ​​that meet the time conditions into the real-time cache layer, ensuring that the freshest data can be processed and accessed quickly. Possible implementation methods include: writing the information representation values ​​and their metadata into a hash table or queue in memory; or, storing them in a dedicated in-memory database with fast read / write capabilities for real-time processing. When the generation time is greater than the first time threshold and the access frequency is greater than a preset frequency threshold, the newly added information representation values ​​are assigned to the recently active layer. The frequency threshold is a preset value used to define the activity level of the data, providing an activity standard for data allocation in the recently active layer. Possible implementation methods include: setting the frequency threshold to, for example, 10 accesses per hour, 50 accesses per day, etc., with the specific value adjusted according to the importance of the data and access patterns; or, the threshold can be dynamically set based on historical data analysis results or business needs. Assigning data to the recently active layer refers to logically or physically placing information representations that meet time and activity criteria into this layer. Its purpose is to ensure that data that, while not the freshest, is frequently accessed, receives local optimization and incremental updates. Possible implementations include: writing the information representations and their index structure to a dedicated HNSW index instance for recently active data; or storing them in a persistent storage area that supports fast incremental updates and local optimization.

[0042] This solution addresses the issue of unreasonable allocation of new data in multi-level, fast-searching indexes for similar information by introducing an intelligent allocation mechanism based on generation time and access frequency. First, the system obtains the generation time of each newly added information representation value to determine its timeliness, and simultaneously obtains its access frequency to assess its activity level. Based on these two key indicators, the system can perform refined data classification. Specifically, for data with extremely short generation times and high immediacy requirements (i.e., generation time less than or equal to a preset first time threshold), the system directly allocates it to the real-time cache layer. As the front-end receiving layer, the real-time cache layer can process this freshest data with extremely low latency, ensuring the system's rapid response capability to real-time information. Furthermore, for data whose generation time has exceeded the first time threshold but still maintains a high access frequency (i.e., generation time greater than the first time threshold and access frequency greater than a preset frequency threshold), the system allocates it to the recently active layer. The recently active layer is specifically designed to handle this type of "recently active" data, allowing for local index optimization to adapt to its dynamically changing access patterns and semantic relationships. This hierarchical allocation strategy ensures that data with varying timeliness and activity levels is placed in the index level best suited to its processing characteristics, thus avoiding the problem of chaotic index structure caused by arbitrarily allocating new data. In this way, the solution ensures that when data enters the fast similar information search index, it is rationally guided to the appropriate level based on its inherent attributes, laying the foundation for subsequent local index optimization and topology self-healing operations. This intelligent hierarchical mechanism based on data characteristics is closely integrated with the overall architecture of building a multi-level fast similar information search index, enabling the real-time caching layer, the recently active layer, and the historically stable layer to each perform their respective functions, jointly maintaining the efficiency and accuracy of the index. It effectively reduces invalid pointers and duplicate information fragments caused by improper data allocation, thereby reducing the risk of internal logical errors in the index and ultimately improving the retrieval efficiency and accuracy of the entire heterogeneous data incremental retrieval index update method.

[0043] In some embodiments, the step of locally indexing newly added information representation values ​​in the recently active layer includes: The quality or distribution characteristics of the newly added information are evaluated to obtain the evaluation results. Based on the evaluation results, adjust the HNSW index construction parameters for quickly finding similar information; HNSW index construction parameters include the number of neighbor connections or the search range to be constructed; Based on the adjusted HNSW index construction parameters, the newly added information representation values ​​are indexed to achieve local index optimization for the newly added information representation values.

[0044] The evaluation process involves assessing the quality and distribution characteristics of newly added information representations. This step aims to gain a deeper understanding of the intrinsic properties of the new data, providing data support for subsequent index parameter adjustments. This evaluation can be achieved by calculating the statistical characteristics of the newly added information representations, such as mean, variance, skewness, and kurtosis, to assess their distribution characteristics. Simultaneously, a pre-trained quality assessment model can be used to score the information representations to determine their quality. Alternatively, clustering algorithms (such as K-means and DBSCAN) can be employed to perform cluster analysis on the newly added information representations, observing their clustering and dispersion in the feature space to evaluate their distribution characteristics. For quality assessment, metrics such as information entropy and signal-to-noise ratio can be used for quantification.

[0045] Specifically: 1. Quality Assessment: For numerical representations of text information, the system can calculate its internal noise level, for example, by statistically analyzing the proportion of garbled characters or the density of low-confidence text recognition results. For numerical representations of image or video information, the system can calculate its sharpness metric (e.g., through image sharpness analysis algorithms) or noise level (e.g., by calculating the signal-to-noise ratio of image or video frames). The calculation results of these metrics constitute the quality assessment result.

[0046] 2. Distribution Characteristic Evaluation: The system can calculate the statistical characteristics of the current batch of information representation values ​​in the information representation space. For example, it can calculate the mean, variance, or covariance matrix of these values ​​to reflect their central tendency and dispersion. Furthermore, the system can calculate the overlap or distance distribution between these values ​​and existing information representation values ​​in the recently active layer to understand the degree of their association with existing data. For example, it can measure the degree of clustering by calculating the average Euclidean distance or cosine similarity of the information representation values ​​within the batch.

[0047] Based on the evaluation results, the HNSW index construction parameters for quickly finding similar information are adjusted. This step dynamically optimizes the HNSW index construction strategy based on an understanding of the characteristics of the new data, ensuring the index structure can adapt to heterogeneous data. A rule-based parameter adjustment strategy can be designed. For example, when the evaluation results show that the new data quality is low, the search range in the HNSW index construction parameters can be appropriately reduced to decrease the impact of noise on the index structure; when the new data distribution is sparse, the number of neighbor connections can be increased to ensure the connectivity of the index. As another implementation approach, machine learning models (such as reinforcement learning or Bayesian optimization) can be used to learn the mapping relationship between the evaluation results and the optimal HNSW index construction parameters. By conducting index construction and retrieval performance tests under different data characteristics, the model can be trained to automatically recommend the appropriate number of neighbor connections or the search range for construction.

[0048] HNSW index construction parameters include the number of neighbor connections and the search range, which directly affect the structure and performance of the HNSW index and are key to index optimization. The number of neighbor connections (M parameter) determines the number of neighbors each node in the HNSW graph connects to during construction, affecting the index density and retrieval efficiency. The search range (efConstruction parameter) determines the size of the neighbor search area during index construction, affecting index quality and construction time. Besides the number of neighbor connections and the search range, HNSW index construction parameters can also include the number of levels (max_level), inter-level join strategies, etc. These parameters collectively determine the topology and retrieval performance of the HNSW index.

[0049] In practical applications, determining the HNSW index building parameters that need adjustment primarily depends on the evaluation results of the "quality or distribution characteristics" of the newly added information representation values. The "number of neighbor connections" and "build search range" parameters in the HNSW index building process have different impacts on the index's structure and performance: Neighbor Connection Count: This parameter determines the number of neighbors each node in the HNSW graph connects to during the construction process. A higher M value makes the index graph denser, potentially improving retrieval recall, but increasing index construction time and memory consumption. A lower M value makes the index graph sparser, speeding up construction and reducing memory usage, but potentially sacrificing recall accuracy. Therefore, when evaluation results show that newly added data is "very dense" in a local area, or its distribution is "complex" requiring stronger connectivity to capture the data structure, it is generally necessary to "increase" the neighbor connection count to ensure index connectivity and retrieval accuracy.

[0050] Search Scope: This parameter controls the number of candidate neighbors the algorithm explores at the current level when building the index. A larger efConstruction value results in a broader search, potentially generating a higher-quality index, but significantly increases build time. A smaller efConstruction value speeds up the index building process but may lead to a decrease in index quality. Therefore, when evaluation results indicate that newly added data is of "low quality" or "contains noise," it is usually necessary to "reduce" the search scope to limit the propagation of noise and ensure that the local structure of the index remains undisturbed, in order to avoid introducing too many invalid connections during index building.

[0051] The correspondence between the evaluation results and the determined parameters to be adjusted, and the adjustment amounts for those parameters, can be quantified and used for decision-making using the following methods: 1. Rule-driven mapping: A series of rules and thresholds can be pre-defined to directly map the quantitative indicators of the evaluation results to the adjustment of HNSW parameters.

[0052] Quality Assessment: If the quality assessment score of a newly added information representation value (e.g., obtained through signal-to-noise ratio, sharpness metrics, or a custom quality model) falls below a certain preset "quality threshold," a rule that reduces the efConstruction parameter can be triggered. For example, when the quality assessment score is below 0.6, efConstruction will be adjusted from the default value of 200 to 100.

[0053] Distribution characteristic assessment: If the distribution characteristic assessment results of the newly added information representing the numerical values ​​(e.g., obtained through cluster density, local density, or overlap with existing data) show that the data is "very dense" in a local area, or forms new clusters that require stronger internal connectivity, a rule to increase the M parameter can be triggered. For example, when the local density indicator is higher than a certain "density threshold", M is adjusted from the default value of 16 to 24.

[0054] These specific values ​​(such as adjusting 200 to 100 and 16 to 24) were determined based on empirical testing and tuning of the HNSW algorithm under specific data types and system environments. They represent parameter values ​​that can effectively optimize index performance under specific evaluation conditions.

[0055] 2. Empirical Tuning and Iterative Testing: In practical applications, multiple experiments can be conducted on representative subsets of data to try different combinations of HNSW parameters. After each experiment, the quality of the constructed index (e.g., recall, query speed) and resource consumption are evaluated to establish an empirical correlation between the evaluation results and the parameter adjustments. For example, A / B testing can be performed on data samples with different quality and distribution characteristics to compare the index performance under different parameter settings, gradually determining the optimal tuning strategy and numerical range.

[0056] 3. Adaptive Learning Mechanism: A more advanced approach is to introduce a lightweight adaptive learning module. This module can learn the mapping relationship between evaluation results and parameter adjustments based on historical evaluation results, parameter tuning records, and final index performance feedback, using a simple machine learning model (such as linear regression or decision trees). For example, the model can learn that when data quality degrades by 10%, efConstruction should be reduced by 50 units.

[0057] Based on the adjusted HNSW index construction parameters, indexing is performed on the newly added information representation values ​​to achieve local index optimization. This step utilizes the optimized parameters to effectively integrate the new data into the recently active layer, forming a locally optimal index structure. In the recently active layer, the HNSW algorithm can be used to construct new HNSW nodes for the newly added information representation values ​​based on the adjusted neighbor connection count and search range, and connect them to the existing index structure of the recently active layer. Simultaneously, the proximity relationships of the newly added information representation values ​​are recalculated, and their connections in the index graph are adjusted. Alternatively, a batch construction approach can be used. A batch of newly added information representation values ​​is treated as a whole, and using the adjusted HNSW index construction parameters, an independent HNSW subgraph is constructed in the recently active layer. This subgraph is then merged with the main index structure of the recently active layer to achieve local index optimization.

[0058] This method first involves constructing a multi-level index for fast similarity information lookup, including a historically stable layer, a recently active layer, and a real-time cache layer. Newly added information representation values ​​are received and transferred in batches from the real-time cache layer to the recently active layer. In the recently active layer, these newly added information representation values ​​undergo local index optimization to achieve incremental updates. To address the issue that newly added information representation values ​​may exhibit significant variations in distribution characteristics due to differences in data source, format, or quality, leading to inconsistencies in the index structure, decreased retrieval efficiency, or reduced accuracy due to the inability of index optimization to adapt to heterogeneous data distributions, this application proposes a solution. Specifically, the process of locally optimizing the index for newly added information representation values ​​in the recently active layer is as follows: First, the quality or distribution characteristics of the newly added information representation values ​​are evaluated to obtain evaluation results. This evaluation provides a data basis for subsequent parameter adjustments, ensuring that the optimization process is targeted. Based on this evaluation result, the HNSW index construction parameters of the fast similarity information lookup index are dynamically adjusted. These parameters include the number of neighbor connections or the search range to be constructed. This dynamic adjustment mechanism allows the index building process to adapt to the heterogeneous characteristics of new data. For example, when data quality is low, the search range can be reduced to mitigate noise; when data is densely distributed, the number of neighbor connections can be increased to improve retrieval accuracy. Finally, based on the adjusted HNSW index building parameters, the newly added information representation values ​​are indexed. This step utilizes optimized parameters to build the index, thereby strengthening the local structural consistency of the index, reducing the risk of invalid pointers or duplicate fragments, and ultimately achieving efficient incremental updates. This method ensures that even with heterogeneous data, the recently active layer maintains an optimized and stable index structure, which is crucial for its subsequent merging into the historical stable layer and the overall performance of quickly finding similar information in the index.

[0059] As a specific implementation, when the system receives a batch of new union activity records, such as video clips, text reports, and images of employees participating in a skills competition, these materials may come from different departments and vary in format and quality. First, the system evaluates the numerical representations of these newly added information. For example, it can calculate the cluster density of the information representations in the feature space for video clips and the entropy of the word vector distribution of the information representations in text reports to determine their distribution characteristics and quality. If the evaluation results show that the distribution of this new data differs significantly from the existing data in the recently active layer, and that the information representation quality of some video clips is low (e.g., due to inaccurate feature extraction caused by low video resolution), the system dynamically adjusts the HNSW index construction parameters based on these evaluation results. Specifically, it may reduce the search range in the HNSW index construction parameters, for example, adjusting the efConstruction parameter from 200 to 100, to avoid introducing too many noisy connections during index construction from low-quality data. Simultaneously, if the new data is very dense in a certain local area, it may appropriately increase the number of neighbor connections, for example, adjusting the M parameter from 16 to 24, to ensure the index connectivity and retrieval accuracy of that area. Subsequently, based on these adjusted parameters, the system recalculates the proximity relationships for the newly added information in the recently active layer, adjusts their connections in the index graph, and constructs the index, thereby achieving local index optimization for this batch of new data. This optimization ensures that the new data can be integrated into the recently active layer in a way that best suits its own characteristics, laying the foundation for subsequent incremental updates and retrieval operations.

[0060] Through the above technical solution, this application effectively addresses the problem that newly added information representation values ​​exhibit significant variations in distribution characteristics due to differences in data source, format, or quality. This leads to index optimization being unable to adapt to heterogeneous data distributions, resulting in inconsistencies within the index structure, decreased retrieval efficiency, or reduced accuracy. Specifically, by evaluating the quality or distribution characteristics of newly added information representation values, the system can gain precise insights into the heterogeneity of new data. Based on this, the HNSW index construction parameters for quickly finding similar information are dynamically adjusted according to the evaluation results. For example, the number of neighbor connections or the search range for construction can be adjusted, allowing the index construction process to fully adapt to the unique distribution and quality characteristics of new data. This targeted parameter adjustment avoids blind optimization and ensures that the index structure maintains its local consistency and stability when facing heterogeneous data. Finally, the index is constructed based on the adjusted parameters for the newly added information representation values, achieving local index optimization for new data in the recently active layer. This significantly improves the efficiency and accuracy of incremental updates and effectively reduces the risk of index structure degradation due to data heterogeneity, thereby ensuring the stability and high performance of the overall retrieval system.

[0061] In some embodiments, the step of analyzing the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer to identify a correlation between their information representation values ​​includes: Calculate the degree of difference between the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer; When the degree of difference is greater than the preset difference threshold, the semantic feature vectors of the original data represented by the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer are extracted respectively, and the similarity between the semantic feature vectors of the original data represented by the two is calculated. When the similarity is greater than the preset semantic consistency threshold, it is determined that the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer belong to a heterogeneous data set with consistent semantic attributes.

[0062] The calculation of the difference between the information representation values ​​in the recently active layer and the historically stable layer aims to preliminarily assess the macroscopic differences in the distribution of information representation values ​​between the two data layers, thereby quickly identifying data groups that may have deep semantic connections but whose surface numerical distributions are inconsistent. This difference can be quantified by calculating the statistical characteristics (such as mean, variance, covariance matrix, KL divergence, JS divergence, etc.) of the information representation value sets in the two data layers. For example, the average vector of all information representation values ​​in the recently active layer and the historically stable layer can be calculated, and the Euclidean distance or cosine distance between these two average vectors can be used as the degree of difference. Alternatively, clustering algorithms (such as K-means, DBSCAN) can be used to cluster the information representation values ​​of the two layers, and then the distance or overlap of the corresponding cluster centers in the two layers can be compared to assess the difference.

[0063] When the difference exceeds a preset threshold, this conditional judgment controls the triggering of subsequent, more computationally intensive semantic analysis. Only when the information representation values ​​of two data layers show significant differences in their macroscopic distribution is further in-depth analysis of their potential semantic relationships necessary. This avoids unnecessary complex calculations on seemingly similar data, improving processing efficiency. This difference threshold can be an empirical value, set according to the actual application scenario and data characteristics. For example, when the Euclidean distance between the average vectors of the information representation values ​​of two layers exceeds a certain value, subsequent analysis is triggered. This difference threshold can also be dynamically adjusted through a machine learning model. This model learns, based on historical data analysis results, the optimal level of difference at which semantic analysis yields the best association recognition results.

[0064] Semantic feature vectors are extracted from the information representation values ​​in the recently active layers and the historically stable layers, respectively. This step aims to extract deeper and more abstract semantic information from the original data represented by the information representation values, overcoming potential heterogeneity, noise, or distribution differences in the original information representation values, thereby more accurately capturing the true meaning of the data. Pre-trained deep learning models (such as BERT, GPT, etc. for text encoders, ResNet, Vision Transformer, etc. for image encoders, or multimodal fusion models) can be used as semantic understanding modules. Original data (such as text, images, and videos) can be input into these models, and the vectors from the last layer or specific intermediate layers of their output can be extracted as semantic feature vectors. For domain-specific data, a domain knowledge graph can also be constructed, and graph embedding techniques (such as TransE, RotatE) can be used to map the information representation values ​​to the entity or relation vector space of the knowledge graph, thereby obtaining feature vectors with domain semantics.

[0065] Calculating the similarity between the semantic feature vectors of the two vectors quantifies the strength of the association between the semantic content represented by the numerical information representations in the recently active layer and the historically stable layer. By comparing the semantic feature vectors, their semantic proximity can be assessed, even if their original numerical information representations differ significantly. Commonly used similarity calculation methods include cosine similarity, which measures the proximity of two vectors along a direction and is suitable for semantic similarity measurement in high-dimensional spaces. Other methods, such as the reciprocal of the Euclidean distance or the Gaussian kernel function, can also be used to calculate similarity; however, these methods may have varying sensitivities to semantic proximity in different scenarios.

[0066] When the similarity exceeds a preset semantic consistency threshold, this conditional judgment is used to ultimately determine whether the information representation values ​​in the two data layers are semantically highly consistent. Only when the semantic similarity reaches a sufficiently high level is it judged as a heterogeneous data set with consistent semantic attributes, thereby ensuring that the subsequent jump edges are based on reliable semantic associations. The semantic consistency threshold can be a fixed value, for example, when the cosine similarity exceeds 0.75 or 0.8, semantic consistency is considered. This value can be optimized based on domain expert knowledge or through methods such as cross-validation. The threshold can also be adaptive, for example, dynamically adjusted according to the overall semantic distribution of the current data batch, or adjusted feedback based on the accuracy of historical association recognition.

[0067] The final output of the analysis process is to determine which information representation values ​​from recently active layers and historically stable layers belong to heterogeneous data sets with consistent semantic attributes. This determination clearly identifies which information representation values ​​from different data layers, while superficially heterogeneous, are highly correlated in deeper meaning. This determination serves as the direct basis for establishing jump edges, ensuring that jump edges connect truly semantically consistent data, thereby improving the accuracy and efficiency of retrieval. Information representation values ​​from recently active layers and historically stable layers that meet the criteria can be paired and stored in an association list, with each pair accompanied by its calculated similarity value. Alternatively, in the metadata of the quick-find similar information index, a "semantic association tag" can be added to these determined information representation values, indicating their semantic consistency with a specific information representation value in another layer.

[0068] This scheme addresses the problem of inaccurate association identification by introducing a multi-step analysis mechanism to ensure more accurate identification of heterogeneous data sets with consistent semantic attributes during association recognition. Specifically, in the heterogeneous data incremental retrieval index update method, to establish effective jump edges between the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer, enabling direct logical addressing across layers, this application proposes a refined association identification mechanism. This mechanism first performs preliminary screening by calculating the degree of difference between the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer. The purpose of this step is to quickly identify data groups with significant differences in the distribution of information representation values, as these data groups are often the most heterogeneous and require the most in-depth semantic analysis. By setting a preset difference threshold, the system can avoid unnecessary complex semantic analysis on data that appears to be highly similar, thereby effectively saving computational resources and improving processing efficiency. When the detected difference degree exceeds the preset difference threshold, the system further initiates the semantic analysis process. At this time, the system extracts the semantic feature vectors corresponding to the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer, respectively. This process is crucial because it extracts more abstract and essential semantic information from raw, potentially heterogeneous numerical representations. Through semantic feature vectors, the system can transcend the superficial formal differences in data and directly capture its deeper meaning. For example, the system can utilize pre-trained semantic understanding modules (such as deep learning-based text understanding models or cross-modal semantic models) to assess the semantic similarity between the original content represented by new and old data (e.g., text descriptions, image content, video events). Subsequently, the system calculates the similarity between these two semantic feature vectors to quantify their semantic proximity. This semantic feature vector-based similarity calculation can more accurately assess the true connections between heterogeneous data, even if they exhibit significant distributional differences in the original information representation numerical space. Finally, by comparing the calculated similarity with a preset semantic consistency threshold, the system can accurately determine whether the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer belong to a heterogeneous data set with consistent semantic attributes. Only when the similarity reaches or exceeds this threshold is the two considered to have sufficient semantic consistency, thus providing a solid foundation for establishing reliable jump edges. This multi-stage, coarse-to-fine analysis method effectively solves the problem of inaccurate association identification caused by large differences in the distribution and inconsistent meanings of heterogeneous data, avoids the erroneous establishment of skip edges, and thus ensures retrieval efficiency and accuracy. Combined with the basic scheme of constructing a multi-level fast similar information index (including a historical stable layer, a recently active layer, and a real-time cache layer) and establishing skip edges in the fast similar information index, this approach can significantly improve the performance of the entire system.In the basic approach, skip edges are established to achieve direct logical addressing across levels. However, if the skip edges connect semantically inconsistent data, they can introduce noise and reduce retrieval accuracy. This approach ensures that the skip edges are high-quality and meaningful connections by accurately identifying heterogeneous data sets with consistent semantic attributes. This means that during the retrieval process, when using skip edges for cross-level addressing, intermediate index levels can be more reliably skipped, directly locating semantically relevant historical data. This significantly improves retrieval efficiency while maintaining high accuracy, effectively addressing the challenges posed by massive amounts of heterogeneous data.

[0069] As a specific implementation method, suppose that in the trade union intelligent question-and-answer system, the recently active layer contains a batch of newly entered employee application materials regarding "innovative achievements." These materials may exist in non-standard formats such as text, images, or videos, and their information representation values ​​differ significantly from the standardized "technological innovation" related files already existing in the historical stable layer. First, the system calculates the degree of difference between the information representation values ​​of these "innovative achievements" in the recently active layer and the information representation values ​​of "technological innovation" in the historical stable layer. For example, the system can calculate the average vector of the two sets of information representation values ​​and find that the Euclidean distance between them is large, indicating a significant difference in numerical distribution. When this degree of difference exceeds a preset difference threshold, the system further initiates semantic analysis. At this point, the system extracts the semantic feature vectors of the information representation values ​​of "innovative achievements" in the recently active layer and the information representation values ​​of "technological innovation" in the historical stable layer, respectively. This can be achieved through a pre-trained cross-modal semantic understanding model that can uniformly encode text, image, and video content into high-dimensional semantic vectors. For example, for a textual description of "innovative achievements," the model generates a vector representing its core semantics; for scanned old documents of "technological innovations," the model also extracts the semantic features of the image content and OCR-recognized text. Next, the system calculates the cosine similarity between these semantic feature vectors. For instance, if the cosine similarity between the semantic vector of a certain "innovative achievement" and the semantic vector of a certain "technological innovation" document is 0.82, and this similarity is greater than a preset semantic consistency threshold (e.g., 0.75), the system determines that the numerical representations of "innovative achievements" in the recently active layer and the numerical representations of "technological innovations" in the historically stable layer belong to a heterogeneous data set with consistent semantic attributes. This means that although they differ in their original data form and the distribution of numerical representations, they are highly correlated in their deeper meaning. Based on this determination, the system can establish a jump edge connecting these two types of numerical representations in the quick similarity search index, allowing subsequent searches to directly jump from a new "innovative achievement" query to the relevant historical "technological innovation" document without traversing intermediate levels.

[0070] Through the above technical solution, this application effectively solves the problem of inaccurate association identification caused by large differences in the distribution and inconsistent meanings of heterogeneous data during the incremental retrieval index update process. By introducing a multi-stage, coarse-to-fine analysis mechanism—firstly, preliminary screening based on the degree of difference, then in-depth extraction of semantic feature vectors and similarity calculation, and finally judgment based on semantic consistency threshold—it ensures that the identified associations are based on reliable semantic consistency. This avoids erroneous jump edges, thus ensuring that the topology of the index for quickly finding similar information accurately reflects the true semantic relationships between data. In practical applications, when the intelligent question-answering system needs to integrate a large amount of historical archive data, even if these materials differ significantly from existing data in format, quality, and numerical distribution, this solution can accurately identify the deep semantic relationships between them and establish high-quality jump edges. This enables the system to perform cross-level addressing more efficiently and accurately during the retrieval process, significantly improving retrieval efficiency and result accuracy, thereby improving the user experience and avoiding situations where additional prompts and confirmation operations are required due to inaccurate information.

[0071] In some embodiments, the step of establishing a jump edge connecting the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer in the fast lookup similar information index includes: Establish jump edges for heterogeneous data sets that share consistent semantic attributes; The association weight is determined based on similarity and assigned to the corresponding jump edge. The association weight is used to indicate the semantic association strength during the retrieval process so that the retrieval process will prioritize the logical path with high semantic association (i.e. high logical priority).

[0072] Specifically, in the fast similar information search index, skip edges are established connecting information representation values ​​in the recently active layer with those in the historically stable layer, aiming to construct a direct logical addressing mechanism across levels. These skip edges can be implemented as logical pointers or references, where an information representation value node in the recently active layer stores a memory address or unique identifier pointing to the corresponding information representation value node in the historically stable layer. Another implementation approach is to define specific edge types for these cross-level associations in the index structure and explicitly store these connections in structures such as graph databases. The core function of these skip edges is to significantly shorten the retrieval path when the retrieval system needs to access data that spans different levels but is semantically highly related, by skipping intermediate index levels.

[0073] Skip edges are established for heterogeneous data sets with consistent semantic attributes, ensuring that only data with deep semantic connections, confirmed through semantic analysis, are directly connected. The establishment of these skip edges can be triggered by an automated semantic analysis program. Once the program identifies semantic consistency between the information representation values ​​in the recently active layer and those in the historically stable layer, it automatically creates the corresponding skip edge. Alternatively, the establishment of skip edges can be controlled through preset rules or thresholds. For example, when the semantic similarity score reaches a specific threshold, the system establishes the corresponding skip edge.

[0074] The association weight is determined based on similarity, aiming to quantify the semantic association strength represented by these jump edges. This association weight can be obtained directly from the semantic similarity score (e.g., a value calculated by cosine similarity), or by performing a function transformation on the original similarity score (e.g., linear scaling or non-linear mapping) to make it more suitable as a priority indicator in the retrieval process.

[0075] Assigning association weights to corresponding jump edges means storing the calculated association weights as an attribute of the jump edge in a fast lookup similarity index. This can be achieved by storing the weights as attributes of edges in the graph structure, or by maintaining a separate mapping table to associate jump edge identifiers with their corresponding weight values.

[0076] Association weights are used to indicate the strength of semantic relevance during the retrieval process, causing the retrieval flow to prioritize logical paths with high semantic relevance (i.e., high logical priority). This means that when performing a retrieval operation, retrieval algorithms (e.g., weighted path search algorithms) use these weights to evaluate the priority of different paths. Jump edges with higher association weights will be assigned higher logical priority, guiding the retrieval system to explore these paths first, thereby locating information that is highly relevant to the query semantics more quickly.

[0077] This application's solution optimizes the retrieval process by establishing jump edges with association weights based on the identification of semantic attribute consistency between information representation values ​​in recently active layers and information representation values ​​in historically stable layers within a multi-level fast similar information index. When the aforementioned analysis steps identify heterogeneous data sets with consistent semantic attributes, this solution establishes direct jump edges for these data sets. These jump edges are assigned association weights determined based on similarity, which quantify the strength of semantic association between data. During the retrieval process, the retrieval system can utilize these association weights as logical priority indicators, prioritizing the exploration of logical paths with high semantic relevance. This mechanism enables the retrieval system to efficiently traverse different index levels and directly access semantically highly related heterogeneous data, avoiding unnecessary intermediate level traversal or inefficient search paths, thereby significantly improving retrieval efficiency and accuracy. In this way, this solution effectively solves the problem of merely identifying associations without efficiently utilizing them for retrieval, enabling the system to provide faster and more accurate responses when processing complex heterogeneous data.

[0078] In some embodiments, the step of: Real-time monitoring of the retrieval frequency and success rate of cross-level addressing via jump edge connections; The association weights are attenuated or enhanced based on the retrieval frequency and retrieval success rate in order to update the logical priority of the jump edge connection. When the updated association weight is lower than the preset path persistence threshold, the corresponding skip edge connection is deleted, and the storage space used to maintain the skip edge connection in the multi-level hierarchical index structure is released.

[0079] The system monitors in real-time the retrieval frequency and success rate of cross-level addressing via jump edge connections, aiming to obtain feedback on the usage and effectiveness of jump edge connections in actual retrieval processes. Retrieval frequency refers to the number of times a specific jump edge is used for cross-level addressing, reflecting its activity level. Retrieval success rate refers to the proportion of relevant information successfully found after addressing via the jump edge, reflecting the effectiveness and accuracy of the jump edge connection. One implementation involves logging each jump edge retrieval, including the jump edge identifier used, the retrieval initiation time, and the retrieval result (e.g., whether the target information was successfully matched). Real-time or periodic analysis of this log data allows for the statistical analysis of the retrieval frequency and success rate of each jump edge. Another implementation maintains a pair of counters for each jump edge: one to record the total number of times it is accessed, and the other to record the number of times it is successfully addressed. During each retrieval operation, these counters are updated accordingly based on whether a jump edge was used and the retrieval results.

[0080] The association weights are attenuated or enhanced based on retrieval frequency and success rate to update the logical priority of jump edge connections. The aim is to dynamically adjust the association weights of jump edges based on their actual performance, thereby updating their logical priority during the retrieval process. Association weight is a numerical measure of the semantic association strength of jump edges, while logical priority determines the retrieval algorithm's preference for selecting jump edges among multiple paths. Attenuation or enhancement can give higher priority to frequently used and high-success-rate jump edges, while lowering the priority of infrequently used or low-success-rate jump edges. One implementation method is to set a series of thresholds and adjustment rules. For example, when both retrieval frequency and success rate are higher than preset thresholds, the association weights are enhanced linearly or non-linearly; when the retrieval frequency is lower than the preset threshold or the success rate drops significantly, the association weights are attenuated. Specific adjustment functions can be designed based on experience or optimization algorithms. Another implementation method is to utilize an adaptive learning mechanism. For example, a simple feedback control system can be built, using retrieval frequency and success rate as inputs, and calculating the adjustment amount of association weights through a preset adjustment strategy (such as a reinforcement learning-based strategy or heuristic rules), thereby achieving dynamic updates of the weights.

[0081] When the updated association weight falls below a preset path persistence threshold, the corresponding skip edge connection is deleted, and the storage space used to maintain that skip edge connection in the multi-level hierarchical index structure is released. This aims to promptly clean up invalid or inefficient skip edge connections to optimize the index structure and save storage resources. The path persistence threshold is a preset minimum association weight value; skip edges below this value are considered to no longer have sufficient validity or value. The deletion operation not only removes logical connections but also reclaims physical storage space, ensuring a streamlined and efficient index structure. One implementation is to immediately check the updated weight value after each association weight update. If the value is below the path persistence threshold, a deletion procedure is triggered to remove the skip edge from the graph structure of the fast lookup similar information index and notify the memory management module to reclaim the memory or disk space it occupies. Another implementation is to set up a background cleanup service that periodically scans all skip edges in the fast lookup similar information index. For skip edges with association weights below the path persistence threshold, this service marks them as pending deletion and performs the deletion operation in the next maintenance cycle, including updating the relevant node pointers and releasing storage resources.

[0082] This solution continuously monitors in real time the usage and effectiveness feedback of each jump edge connection in actual retrieval operations. Retrieval frequency reflects the activity level of the jump edge utilized by the query system, while retrieval success rate directly measures the accuracy with which it guides valid results. This monitoring data provides an objective basis for subsequent dynamic adjustments. Based on this real-time monitoring data, the system will attenuate or enhance the association weight of jump edges according to a preset strategy. For example, for jump edges that are frequently used and have a high retrieval success rate, their association weight will be enhanced, thereby increasing their logical priority in retrieval path selection, making subsequent retrievals more inclined to utilize these efficient connections. Conversely, for jump edges that are used infrequently or have a poor retrieval success rate, their association weight will be attenuated, reducing their logical priority. This dynamic adjustment mechanism ensures that the weight of a jump edge always reflects its current actual value and effectiveness, avoiding the problem of weight invalidation due to data evolution. Furthermore, to prevent invalid or inefficient jump edges from occupying system resources for a long time and interfering with retrieval efficiency, this application sets a path persistence threshold. When the association weight of a skip edge falls below a preset threshold after decay processing, the system determines that the skip edge has lost its value and immediately performs a deletion operation. The deletion operation not only removes the skip edge from the logical structure of the fast similar information search index, but also simultaneously releases the storage space used to maintain the skip edge connection in the multi-level hierarchical index structure.

[0083] In some embodiments, the step of: When merging the information representation values ​​in the recently active layer into the historical stable layer, cluster analysis is performed on the information representation values ​​in the recently active layer to obtain cluster centers. Cluster centers that meet the preset stability conditions and their associated information representation values ​​are merged into the historical stable layer. The associated information representation values ​​refer to all data points in the cluster corresponding to the cluster center.

[0084] Specifically, the timing of merging information representation values ​​from the recently active layer into the historical stable layer refers to the period before or during the system's data migration operation, where the pre-processed and locally optimized information representation values ​​from the recently active layer are prepared to be integrated into the core index structure of the historical stable layer for long-term storage and retrieval. Its purpose is to ensure that further quality control and structural optimization are possible before the data enters the core stable layer, maintaining the stability and efficiency of the entire fast similar information search index. This operation can be triggered periodically by the system, such as every fixed time period (e.g., hourly or daily), or automatically triggered when the amount of data in the recently active layer reaches a preset threshold. Cluster analysis is then performed on the information representation values ​​in the recently active layer. Cluster analysis is an unsupervised learning method that aims to group objects in a dataset into several groups (clusters) based on their similarity, ensuring high similarity within the same group and low similarity between different groups. In this context, its role is to identify the inherent structure and distribution patterns of the information representation values ​​in the recently active layer, grouping data points with similar semantics or features into one category, thus providing a basis for subsequent stability assessment. Cluster analysis can be implemented using various algorithms. For example, the K-means algorithm can be used to iteratively optimize and assign data points to K clusters; or the DBSCAN algorithm can be used to group data points based on their density, identifying clusters with arbitrary shapes. Cluster analysis yields cluster centers, which are representative points in the cluster analysis results, typically the geometric center of each cluster (e.g., the mean vector) or the most representative data point. Their role is to simplify the evaluation of the entire cluster; by analyzing a few cluster centers, one can indirectly understand the overall characteristics of the large amount of information they represent, thereby improving processing efficiency. A cluster center can be the average vector of all information representations within a cluster, or the information representation value within the cluster that is closest to all other points. Subsequently, cluster centers that meet preset stability conditions and their associated information representation values ​​are merged into the historical stable layer. This step means that after cluster analysis, the system evaluates each cluster center according to pre-defined criteria; only those cluster centers determined to be "stable" and their contained information representation values ​​are allowed to migrate to the historical stable layer. Its function is to act as a quality threshold, ensuring that only high-quality, high-confidence data enters the core index, thereby maintaining the purity and reliability of the historical stable layer. Preset stability conditions may include, but are not limited to: the mean similarity between data points within a cluster being higher than a certain threshold; the similarity between the cluster center and existing data in the historical stable layer being higher than a certain threshold; or the proportion of outliers in a cluster being lower than a certain threshold. The above operations aim to prevent unstable data from impacting the index structure of the historical stable layer. This objective refers to proactively identifying and isolating data that may negatively affect the historical stable layer through the aforementioned clustering analysis and stability screening mechanisms.Unstable data may include low-quality, noisy, abnormally distributed, or semantically inconsistent information representations with existing historical data. By preventing this data from directly entering the historical stable layer, we can prevent it from disrupting the established index structure of the historical stable layer, reduce invalid pointers and duplicate information fragments, and thus ensure retrieval efficiency and accuracy.

[0085] This solution introduces a data stability assessment mechanism to manage the information representation values ​​to be merged in a refined manner during the critical migration process from the recently active layer to the historically stable layer. Specifically, when the system prepares to integrate information representation values ​​from the recently active layer into the historically stable layer, it first performs cluster analysis on all information representation values ​​in the recently active layer. This process aims to automatically divide the information representation values ​​into several logically closely related data clusters based on their inherent characteristics and similarities. Through cluster analysis, the system can identify representative points of each data cluster, i.e., cluster centers. These cluster centers, as abstract generalizations of their respective clusters, can efficiently reflect the overall characteristics of the cluster. Subsequently, the system rigorously evaluates these cluster centers according to preset stability conditions. Only those cluster centers that are determined to meet the stability criteria, and all their associated information representation values, are allowed to be merged into the historically stable layer. This selective merging strategy ensures that only verified, high-quality information representation values ​​that are compatible with the existing data structure of the historically stable layer can enter the core index. Through the aforementioned mechanism, this solution effectively addresses the problem of unstable data impact caused by differences in the distribution of old and new data and inconsistencies in the meaning of information in the information database during the incremental retrieval index update process for heterogeneous data. In the basic heterogeneous data incremental retrieval index update method, the index for quickly finding similar information is constructed as a multi-level structure, including a historically stable layer, a recently active layer, and a real-time cache layer, and implements incremental updates of newly added information representation values ​​and topological self-healing of the index. Building upon this, this solution further enhances the stability of the historically stable layer. By performing cluster analysis and stability screening before merging, this solution can proactively filter out unstable data that may lead to index structure chaos, invalid pointers, or duplicate information fragments. This not only maintains the integrity and consistency of the historical stable layer index structure, but also avoids the system rollback to a time-consuming one-by-one comparison and search mechanism triggered by data quality issues. This ensures the retrieval efficiency and accuracy of the intelligent question answering system when processing massive amounts of heterogeneous data. Especially when integrating complex data such as historical "model worker" deeds archives, it can effectively cope with challenges such as the complexity of data sources and formats, differences in the distribution of information representation values, and inconsistencies in meaning, ensuring that the system can continuously provide high-confidence query results and maintain a second-level response capability.

[0086] As a specific implementation method, when merging information representation values ​​from the recently active layer into the historical stable layer, the system can first perform a K-means clustering algorithm on the information representation values ​​to be merged in the recently active layer. For example, K can be set to 100, dividing all information representation values ​​to be merged into 100 clusters, and calculating the centroid of each cluster as the cluster center. Subsequently, the system can define stability conditions for each cluster center. For example, a stability condition could be: the average cosine similarity between all information representation values ​​within the cluster represented by the cluster center and the cluster center itself must be higher than 0.85, and the average cosine similarity between the cluster center and its 10 nearest neighbors in the historical stable layer must also be higher than 0.70. Only cluster centers that simultaneously meet both conditions are considered stable. For these stable cluster centers, the system will batch merge all associated information representation values ​​(i.e., all data points within the cluster) into the historical stable layer. During the merging process, the batch insertion function of the HNSW index can be used to ensure that this stable data can be efficiently integrated into the index structure of the historical stable layer. For cluster centers and their associated information representation values ​​that do not meet the stability conditions, the system can choose to temporarily retain them in the recently active layer for further observation, reprocessing, or mark them for manual review, thereby avoiding the impact of these potentially unstable data on the index structure of the historical stable layer.

[0087] In some embodiments, the step of merging cluster centers that meet preset stability conditions and their associated information representation values ​​into the historical stability layer includes: Before performing the merge operation, a temporary bidirectional mapping table is established between the recently active layer and the historical stable layer for the information representation values ​​to be migrated; During the merging operation, if a retrieval request for the numerical representation of the information to be migrated is received, the retrieval instruction is synchronously distributed to the recently active layer and the historical stable layer through a temporary bidirectional mapping table, and the returned retrieval results are deduplicated and merged. After the merging operation is completed and the historical stable layer sends a successful write signal, the temporary bidirectional mapping table is revoked, and subsequent retrieval instructions for the numerical representation of the information to be migrated are directed to the historical stable layer.

[0088] A temporary bidirectional mapping table is a key data structure that provides a dynamic and queryable association mechanism during the migration of data from the recently active layer to the historical stable layer. This mapping table records the current position or index of the information representation value to be migrated in both layers, allowing the system to simultaneously track and access this data during the merge process. Specifically, this mapping table can be implemented in various forms. For example, it can be an in-memory hash table where the key is a unique identifier for the information representation value, and the value is a structure containing the corresponding index or address in the recently active layer and the historical stable layer; alternatively, it can be a lightweight key-value store, such as Redis, to store these dynamic mapping relationships for efficient read and write operations. Synchronous distribution of retrieval commands means that during the merge operation, when the system receives a retrieval request for the information representation value to be migrated, the request is simultaneously sent to both the recently active layer and the historical stable layer. This mechanism aims to ensure timely responses regardless of the data's current layer during the merge process (i.e., not yet fully migrated to the historical stable layer or migrated but still retained as a copy in the recently active layer). In implementation, this can be handled by a dedicated request dispatcher. After querying the temporary bidirectional mapping table, the dispatcher copies the original search request and sends it in parallel to the search service interfaces of the recently active layer and the historically stable layer. Alternatively, the application logic layer can trigger search operations on both layers simultaneously through multithreading or asynchronous tasks. Deduplication and merging refers to the system receiving the search results from the recently active layer and the historically stable layer after synchronously distributing the search instructions, and then integrating and optimizing these results. The purpose is to eliminate duplicate data that may result from the two-layer search and ensure that the search results presented to the user are unique, accurate, and consistent. Specifically, this can be achieved by comparing unique identifiers (e.g., data IDs or content hash values) representing numerical information to remove duplicates; or, if duplicate results exist, filtering can be performed according to a preset priority strategy (e.g., prioritizing results from the historically stable layer, or selecting the latest result based on the data version number). Revoking the temporary bidirectional mapping table means that after the data merging operation is successfully completed and the historically stable layer has confirmed that the data was written correctly and returned a success signal, the system removes the mapping table from the runtime environment. This operation aims to clean up temporary resources, preventing the system from continuing to rely on outdated mapping relationships, thereby simplifying subsequent retrieval logic and improving system efficiency. Implementation methods may include releasing the memory space occupied by the mapping table or deleting the corresponding mapping record in persistent storage. Directing to the historically stable layer means that after the temporary bidirectional mapping table is revoked, all subsequent retrieval requests for values ​​representing migrated information will be directly directed to the historically stable layer. This mechanism ensures the uniformity and determinism of data access paths; once data is deemed stable and assigned to the historically stable layer, all queries will be performed directly on that layer, thereby optimizing retrieval efficiency and maintaining data consistency.In practice, the query target representing a specific information value can be permanently pointed to the historical stable layer by updating the system's routing configuration or data location service.

[0089] This application aims to address the potential retrieval interruption issue that may occur when merging information representation values ​​from the recently active layer to the historically stable layer, thereby ensuring retrieval continuity during data migration. This solution achieves this goal through a series of coordinated steps. First, before performing the merge operation, the system establishes a temporary bidirectional mapping table between the recently active layer and the historically stable layer for the information representation values ​​to be migrated. This mapping table serves as a dynamic index during data migration, recording the corresponding position of each piece of data to be migrated in both layers. Its establishment allows the system to simultaneously identify and access information in both the old position (recently active layer) and the new position (historically stable layer) after the merge begins, laying the foundation for subsequent retrieval continuity. Second, during the merge operation, if the system receives retrieval requests for these information representation values ​​to be migrated, it will not reject or delay the requests because data migration is underway. Instead, the system utilizes the previously established temporary bidirectional mapping table to synchronously distribute retrieval instructions to both the recently active layer and the historically stable layer. This synchronous distribution mechanism ensures that logical accessibility is not interrupted regardless of the physical layer in which the data is located. Since results may be returned from both levels, the system performs deduplication and merging of the returned search results to eliminate redundancy and provide a unified and accurate search view. Finally, once the merging operation is successfully completed and the historical stable layer sends a successful write signal, the system immediately cancels the previously established temporary bidirectional mapping table. This action aims to clean up temporary resources and marks the formal end of the data migration. Simultaneously, all subsequent search instructions targeting these migrated information representation values ​​will be directed to the historical stable layer. This not only ensures the uniformity of the data access path but also allows the historical stable layer to fully leverage its advantages as a stable data storage layer, providing efficient and consistent search services. Through this mechanism, the scheme in this application, combined with the scheme described above that merges cluster centers and their associated information representation values ​​that meet preset stability conditions into the historical stable layer, forms a seamless data migration and retrieval continuity assurance system. In the above scheme, cluster analysis and stability condition screening ensure that only high-quality, stable data is merged into the historical stable layer, avoiding the impact of unstable data on the historical stable layer's index structure. Building upon this foundation, the solution proposed in this application further addresses the critical issue of maintaining uninterrupted retrieval services during the migration of data from a recently active layer to a historically stable layer. By providing a "dual-active" retrieval path during the migration and smoothly switching to the historically stable layer after migration, it ensures the continuity of retrieval and the stability of the user experience throughout the entire data lifecycle, effectively avoiding retrieval interruptions or data inconsistencies caused by data migration.

[0090] As a specific implementation method, merging information representation values ​​from the recently active layer to the historical stable layer can be performed using the following steps: Before the merge operation begins, the system can first create an in-memory hash table as a temporary bidirectional mapping table. For each information representation value about to migrate from the recently active layer to the historical stable layer, the system uses its unique identifier as the key and stores a structure containing its current index position in the recently active layer and its pre-allocated index position in the historical stable layer as the value. For example, for the information representation value "A", the mapping table can record its index in the recently active layer as "RA_idx" and its index in the historical stable layer as "HA_idx". During the merge operation, suppose a user initiates a retrieval request for the information representation value "A". At this time, the system's retrieval service will first query the temporary bidirectional mapping table. Since the information representation value "A" is being migrated, its index information exists in both layers of the mapping table. The retrieval service will immediately copy the retrieval request and send it simultaneously to the retrieval interface of the recently active layer (querying "RA_idx") and the retrieval interface of the historical stable layer (querying "HA_idx"). Once both layers return search results, the search service compares the results based on the unique identifier of the information representation value and removes duplicates. For example, if both the recently active layer and the historically stable layer return detailed information representing the value "A", the system will retain only one copy and present it to the user. Once the historically stable layer successfully completes the write operation of the information representation value "A" and sends a success signal to the system, the system immediately deletes the mapping record corresponding to the information representation value "A" from the temporary bidirectional mapping table in memory. Afterward, any search request for the information representation value "A" will no longer query the mapping table but will be directly routed to the historically stable layer for processing. In this way, it is ensured that the user's retrieval of information representation values ​​remains continuous and imperceptible throughout the entire data migration process.

[0091] Through the above technical solution, this application effectively solves the problem of retrieval interruption that may occur when data is merged from the recently active layer to the historical stable layer during the incremental retrieval index update process of heterogeneous data. Specifically, by establishing a temporary bidirectional mapping table before merging, the system can provide a "dual-active" retrieval path for the numerical representation of information to be migrated during data migration, ensuring that retrieval requests are responded to in a timely manner regardless of the data's level. Synchronously distributing retrieval instructions and performing deduplication and merging processes avoids retrieval failures or data inconsistencies caused by uncertain data locations. After merging, the mapping table is promptly revoked and retrieval instructions are redirected to the historical stable layer, which not only cleans up temporary resources but, more importantly, ensures the uniformity of data access paths and optimizes retrieval efficiency. Therefore, the solution of this application significantly improves the continuity of retrieval during the data merging process, ensuring that users can still obtain a stable and seamless retrieval experience during data updates, thereby avoiding the negative impact of internal system maintenance operations on normal user operation.

[0092] This invention provides a heterogeneous data incremental retrieval index update system (this heterogeneous data incremental retrieval index update system adopts the heterogeneous data incremental retrieval index update method of the above embodiments, and the specific process is described in the corresponding steps above), including: Module 100 is used to build a multi-level index for fast similar information lookup; the fast similar information lookup index includes a historical stable layer, a recently active layer, and a real-time cache layer; The update module 200 is used to receive newly added information representation values, and to transfer the newly added information representation values ​​from the real-time cache layer to the recently active layer in batches. In the recently active layer, the local index optimization is performed on the newly added information representation values ​​to achieve incremental updates of the newly added information representation values. The association module 300 is used to analyze the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer before merging the information representation values ​​in the recent active layer into the historical stable layer, so as to identify the relationship between the information representation values ​​of the two, and to establish a jump edge connecting the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer in the fast search similar information index. The sorting module 400 is used to identify invalid pointers and duplicate information fragments in the quick lookup similar information index, and delete invalid pointers and merge duplicate information fragments to realize the topology self-healing operation of the quick lookup similar information index.

[0093] In some embodiments, a monitoring module 500 is also included for performing the following steps: Real-time monitoring of the retrieval frequency and success rate of cross-level addressing via jump edge connections; The association weights are attenuated or enhanced based on the retrieval frequency and retrieval success rate in order to update the logical priority of the jump edge connection. When the updated association weight is lower than the preset path persistence threshold, the corresponding skip edge connection is deleted, and the storage space used to maintain the skip edge connection in the multi-level hierarchical index structure is released.

[0094] In some embodiments, an analysis module 600 is also included for performing the following steps: When merging the information representation values ​​in the recently active layer into the historical stable layer, cluster analysis is performed on the information representation values ​​in the recently active layer to obtain cluster centers. Cluster centers that meet the preset stability conditions and their associated information representation values ​​are merged into the historical stable layer. The associated information representation values ​​refer to all data points in the cluster corresponding to the cluster center.

[0095] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0096] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for updating an incremental retrieval index for heterogeneous data, characterized in that, Includes the following steps: Build a multi-level index for fast searching of similar information; The index for quickly finding similar information includes a historical stable layer, a recently active layer, and a real-time cache layer; The system receives newly added information representation values ​​and transfers them in batches from the real-time cache layer to the recently active layer. In the recently active layer, the system performs local index optimization on the newly added information representation values ​​to achieve incremental updates of the newly added information representation values. Before merging the information representation values ​​in the recently active layer into the historical stable layer, the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer are analyzed to identify the correlation between the two information representation values, and a jump edge connecting the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer is established in the fast search similar information index. Identify invalid pointers and duplicate information fragments in the fast lookup similar information index, delete invalid pointers and merge duplicate information fragments to achieve topological self-healing operation of the fast lookup similar information index.

2. The heterogeneous data incremental retrieval index update method according to claim 1, characterized in that, The steps of receiving newly added information representation values ​​and transferring them in batches from the real-time cache layer to the recently active layer include: The newly added information represents the generation time and access frequency of the value; When the generation time is less than or equal to the preset first time threshold, the newly added information representation value will be allocated to the real-time cache layer; When the generation time is greater than the first time threshold and the access frequency is greater than the preset frequency threshold, the newly added information will be assigned to the recently active layer.

3. The heterogeneous data incremental retrieval index update method according to claim 1, characterized in that, The steps for local indexing optimization of newly added information representation values ​​in the recently active layer include: The quality or distribution characteristics of the newly added information are evaluated to obtain the evaluation results. Based on the evaluation results, adjust the HNSW index construction parameters for quickly finding similar information. Based on the adjusted HNSW index construction parameters, the newly added information representation values ​​are indexed to achieve local index optimization for the newly added information representation values.

4. The heterogeneous data incremental retrieval index update method according to claim 3, characterized in that, HNSW index building parameters include the number of neighbor connections or the search range to build.

5. The heterogeneous data incremental retrieval index update method according to claim 1, characterized in that, The steps for analyzing the information representation values ​​in the recently active layer and the historically stable layer to identify the correlation between their information representation values ​​include: Calculate the degree of difference between the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer; When the degree of difference is greater than the preset difference threshold, the semantic feature vectors of the original data represented by the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer are extracted respectively, and the similarity between the semantic feature vectors of the original data represented by the two is calculated. When the similarity is greater than the preset semantic consistency threshold, it is determined that the information representation values ​​in the recently active layer and the information representation values ​​in the historical stable layer belong to a heterogeneous data set with consistent semantic attributes.

6. The heterogeneous data incremental retrieval index update method according to claim 5, characterized in that, The steps for establishing a jump edge connecting the information representation values ​​in the recently active layer and the information representation values ​​in the historically stable layer in the fast similar information search index include: Establish jump edges for heterogeneous data sets that share consistent semantic attributes; The association weights are determined based on similarity, and the association weights are assigned to the corresponding jump edges.

7. The heterogeneous data incremental retrieval index update method according to claim 6, characterized in that, It also includes the following steps: Real-time monitoring of the retrieval frequency and success rate of cross-level addressing via jump edge connections; The association weights are attenuated or enhanced based on the retrieval frequency and retrieval success rate in order to update the logical priority of the jump edge connection. When the updated association weight is lower than the preset path persistence threshold, the corresponding skip edge connection is deleted, and the storage space used to maintain the skip edge connection in the multi-level hierarchical index structure is released.

8. The heterogeneous data incremental retrieval index update method according to claim 1, characterized in that, It also includes the following steps: When merging the information representation values ​​in the recently active layer into the historical stable layer, cluster analysis is performed on the information representation values ​​in the recently active layer to obtain cluster centers. Cluster centers that meet the preset stability conditions and their associated information representation values ​​are merged into the historical stable layer. The associated information representation values ​​refer to all data points in the cluster corresponding to the cluster center.

9. The heterogeneous data incremental retrieval index update method according to claim 8, characterized in that, The steps of merging cluster centers that meet preset stability conditions and their associated information representations into the historical stability layer include: Before performing the merge operation, a temporary bidirectional mapping table is established between the recently active layer and the historical stable layer for the information representation values ​​to be migrated; During the merging operation, if a retrieval request for the numerical representation of the information to be migrated is received, the retrieval instruction is synchronously distributed to the recently active layer and the historical stable layer through a temporary bidirectional mapping table, and the returned retrieval results are deduplicated and merged. After the merging operation is completed and the historical stable layer sends a successful write signal, the temporary bidirectional mapping table is revoked, and subsequent retrieval instructions for the numerical representation of the information to be migrated are directed to the historical stable layer.

10. A heterogeneous data incremental retrieval index update system, characterized in that, include: Build modules are used to construct multi-level indexes for quickly finding similar information; The index for quickly finding similar information includes a historical stable layer, a recently active layer, and a real-time cache layer; The update module is used to receive newly added information representation values, and to transfer the newly added information representation values ​​from the real-time cache layer to the recently active layer in batches. In the recently active layer, the local index optimization is performed on the newly added information representation values ​​to achieve incremental updates of the newly added information representation values. The association module is used to analyze the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer before merging the information representation values ​​in the recent active layer into the historical stable layer, so as to identify the relationship between the information representation values ​​of the two, and to establish a jump edge connecting the information representation values ​​in the recent active layer and the information representation values ​​in the historical stable layer in the quick search similar information index. The sorting module is used to identify invalid pointers and duplicate information fragments in the quick lookup similar information index, and delete invalid pointers and merge duplicate information fragments to achieve topological self-healing operation of the quick lookup similar information index.

Images