Method and system for fast retrieval of multi-hop relationships in data warehouse based on graph embedding index

Abstract

This invention discloses a fast method and system for multi-hop relationship retrieval in data warehouses based on graph embedding indexes, belonging to the field of multi-hop relationship retrieval data processing technology. The fast method for multi-hop relationship retrieval in data warehouses based on graph embedding indexes includes: data flow interference monitoring; index update timeliness monitoring; multi-hop relationship query accuracy monitoring; and data query response timeliness monitoring. This invention determines whether to perform graph modeling to obtain a graph model by judging whether an interference suppression mechanism is triggered. Secondly, it constructs a multi-level index based on the graph model, and determines whether to trigger an index step size update mechanism based on the index update timeliness judgment result. Finally, it determines whether to trigger a candidate node filtering range narrowing mechanism based on the multi-hop relationship result query accuracy verification result. This achieves the effect of improving the efficiency of multi-hop relationship retrieval in data warehouses and solves the problem of low efficiency in multi-hop relationship retrieval in data warehouses due to data flow interference in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of multi-hop relationship retrieval data processing technology, and in particular to a fast multi-hop relationship retrieval method and system for data warehouses based on graph embedding indexes. Background Technology

[0002] In the process of data retrieval from a data warehouse based on graph embedding indexes, the process begins by capturing real-time data from business sources (such as social network interactions) using data acquisition tools (e.g., Kafka, Flume). This data includes streaming log data, structured form data, and unstructured file data. The collected data (e.g., user clickstream logs) is then stored in the data warehouse. Next, data analysis techniques (e.g., static analysis) are used to transform the table structure, foreign key relationships, and dimensional associations in the data warehouse into a directed heterogeneous graph. Subsequently, graph embedding algorithms (e.g., Node2Vec, DeepWalk) are used to generate low-dimensional embedding vectors for nodes and edges. Combined with indexing strategies such as HNSW (Hierarchical Navigable Small World), a hierarchical index structure is constructed. When a user submits a multi-hop query, the data query center first processes the query through a parser, such as SQL (Structured Query Language). The system uses parsers such as the Structured Query Language (SCL) to convert query statements into graph path expressions, decompose complex queries into atomic subqueries and generate corresponding embedding vector query conditions. It also employs an active node marking mechanism to iteratively expand neighboring nodes, combined with subgraph partitioning techniques to reduce cross-subgraph computation. Distributed storage and query systems, such as ElasticSearch and HDFS (Hadoop Distributed File System), are then used for data storage and retrieval. Spark aggregates the results and converts them into structured data for return. Finally, when changes occur in the data warehouse, retrieval techniques (such as MerkleTree) are used to detect changes, and dynamic metadata tagging technology is used to regenerate embedding vectors and update indexes only for changed nodes to maintain consistency.

[0003] In the process of data retrieval in a data warehouse, existing technologies first analyze the database table types, column names, and inter-table constraints to distinguish between data tables storing specific objects and relational tables storing relationships. Next, each data table is mapped to a concept node, and each record in the table generates a corresponding entity node, establishing an "include" relationship between concept nodes and entity nodes. Subsequently, relationships between entities are constructed based on foreign key constraints or relational tables. Then, a knowledge graph is stored in a graph database (such as Google Cayley) and a Gremlin syntax query interface is provided. When a user inputs a natural language query, the data query center first converts the query into a graph path expression through semantic parsing, then uses Gremlin to perform multi-hop relationship retrieval path matching for entities in the knowledge graph, and finally returns the retrieval results in structured data format.

[0004] For example, Chinese Invention Patent Announcement CN116108238B discloses an optimization method, system, and apparatus for multi-hop queries in a graph database. The method includes: First, parsing the multi-hop query statement in the graph database, extracting key information such as the starting node and relation type, and generating an initial query plan based on the graph database's metadata to determine candidate query path directions; next, optimizing the initial query plan based on the distribution characteristics of nodes and relations in the graph database, and selecting efficient query paths by calculating the selectivity estimates of different paths; then, dynamically monitoring the query progress and intermediate result size during multi-hop query execution, and triggering a pruning strategy when the intermediate result set is detected to be too large, removing candidate nodes or paths with low relevance; finally, performing consistency verification and deduplication on the results returned by the multi-hop query to ensure that the results meet the query semantics and accuracy requirements, and caching the optimized query plan in a data warehouse.

[0005] For example, Chinese invention patent announcement CN117556086B discloses a multi-hop path query method, apparatus, computer device, and storage medium, which includes: firstly, processing the raw data to construct a graph data structure containing nodes, edges, and edge attributes, and establishing an auxiliary index; then receiving a query request containing parameters such as the starting node, ending node, maximum hop count, and edge attribute constraints, and initializing a search queue, a set of visited nodes, and a hop count counter based on the starting node; subsequently, iteratively expanding the path, traversing the neighbors of the current node that meet the constraints to generate a new path, while applying a pruning strategy to remove invalid paths that exceed the hop count limit, contain loops, or are impossible to reach the endpoint, and prioritizing the expansion of paths that are more likely to reach the endpoint; then, performing constraint verification and deduplication filtering on the searched endpoint path; finally, sorting the paths that meet the conditions according to the rules and returning the results.

[0006] The above-mentioned technology has at least the following technical problems:

[0007] During data warehouse data search, noise data from multi-source data streams (such as redundant information reported repeatedly and abnormal data with disordered formats) and sudden peak flows (such as dense alarm data when equipment fails) may cause data stream interference, leading to deviations in data cleaning and standardization, and entity relationships being misled by false interactions. Consequently, the timeliness and accuracy of graph index construction are significantly reduced when updating graph indexes based on dynamic data. In complex multi-hop relationship queries, it is difficult to cope with dynamic data streams and non-linear retrieval requirements, resulting in low path traversal efficiency and query response delays. There is a problem of low efficiency in multi-hop relationship retrieval in data warehouses due to data stream interference. Summary of the Invention

[0008] To address the problem of low retrieval efficiency for multi-hop relationships in data warehouses due to data flow interference in existing technologies, this invention provides a fast retrieval method and system for multi-hop relationships in data warehouses based on graph embedding indexes. The technical solution is as follows:

[0009] On the one hand, a fast retrieval method for multi-hop relationships in data warehouses based on graph embedding indexes is provided. This method includes: during data warehouse data search, determining whether to trigger an interference suppression mechanism based on data flow interference judgment results; after passing the data flow interference judgment, performing graph modeling based on the acquired qualified preprocessed data to obtain a graph model, where qualified preprocessed data represents preprocessed data obtained after cleaning and standardization, and preprocessed data represents collected multi-source data; after graph modeling is completed, constructing a multi-level graph index based on the qualified preprocessed data and the graph model, and judging the timeliness of index updates based on real-time changes in dynamic data flow, determining whether to trigger an index step size update mechanism based on the index update timeliness judgment results; after passing the index update timeliness judgment, performing multi-hop relationship queries in the data warehouse based on the qualified multi-level graph index, and determining whether to trigger a candidate node filtering range narrowing mechanism based on the multi-hop relationship query accuracy verification results; after passing the multi-hop relationship query accuracy verification, determining whether to trigger a query response optimization mechanism based on the data query response timeliness judgment results.

[0010] On the other hand, a fast multi-hop relationship retrieval system for data warehouses based on graph embedding indexes is provided. This system applies methods such as the fast multi-hop relationship retrieval method for data warehouses based on graph embedding indexes. The system includes: a data flow interference monitoring module, an index update timeliness monitoring module, a multi-hop relationship query accuracy monitoring module, and a data query response timeliness monitoring module. Specifically, the data flow interference monitoring module determines whether to trigger an interference suppression mechanism based on the data flow interference judgment result. After the data flow interference judgment is qualified, a graph model is obtained by modeling a graph based on the acquired qualified preprocessed data. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization, and the preprocessed data represents the collected multi-source data. The index update timeliness... The performance monitoring module is used to construct a multi-level graph index based on qualified preprocessed data and the graph model after graph modeling is completed, and to judge the timeliness of index updates based on real-time changes in dynamic data flow. Based on the timeliness judgment result, it decides whether to trigger the index step size update mechanism. The multi-hop relationship query accuracy monitoring module is used to perform multi-hop relationship queries on the data warehouse based on qualified multi-level graph index after the index update timeliness judgment is qualified, and to decide whether to trigger the candidate node screening range shrinking mechanism based on the multi-hop relationship query accuracy verification result. The data query response timeliness monitoring module is used to decide whether to trigger the query response optimization mechanism based on the data query response timeliness judgment result after the multi-hop relationship query accuracy verification is qualified.

[0011] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0012] 1. An interference suppression mechanism is triggered based on the data flow interference assessment results. This mechanism improves the accuracy of overall data processing and analysis. After the data flow interference assessment is passed, a graph model is created based on the acquired qualified preprocessed data. This helps reduce the interference of noisy data and sudden peak flows on data cleaning, standardization, and entity relationships, laying a reliable data foundation for subsequent processing. Qualified preprocessed data refers to the preprocessed data obtained after cleaning and standardization, representing multi-source data. After graph modeling, a multi-level graph index is built based on the qualified preprocessed data and the graph model. This multi-level graph index enhances the ability to quickly retrieve and accurately match multi-hop relationship queries in the data warehouse. The timeliness of index updates is assessed based on real-time changes in the dynamic data flow. The timeliness of index updates is then assessed to determine whether to trigger an index step size update mechanism. This helps improve the timeliness and accuracy of graph index construction, ensuring it can adapt to dynamic data flow changes. The step-size update mechanism is used to improve the real-time synchronization of multi-level graph indexes with dynamic data streams. After the index update timeliness judgment is qualified, multi-hop relationship queries are performed on the qualified multi-level graph index. The candidate node filtering range shrinking mechanism is triggered based on the query accuracy verification result of the multi-hop relationship results. This helps to enhance the ability of complex multi-hop relationship queries to cope with dynamic data streams and non-linear retrieval needs, improve path traversal efficiency, and improve the matching degree between multi-hop relationship results and query targets. After the multi-hop relationship result query accuracy verification is qualified, the query response optimization mechanism is triggered based on the data query response timeliness judgment result. This helps to reduce data query response latency and improve the multi-hop relationship retrieval efficiency of the data warehouse. The query response optimization mechanism is triggered to improve the real-time performance of data query processing. This improves the multi-hop relationship retrieval efficiency of the data warehouse and effectively solves the problem of low multi-hop relationship retrieval efficiency in the data warehouse due to data stream interference in the existing technology.

[0013] 2. By harmonic averaging the data stream interference items to obtain the data stream interference index, it is helpful to comprehensively consider the combined impact of three key interference factors within the window: noisy data, sudden peak flows, and peak data. This avoids bias caused by single-parameter evaluation and makes the data stream interference index more objectively reflect the overall degree of influence of noise and peak flows on the preprocessed data. Based on the obtained data stream interference index, it is possible to determine whether to trigger the interference suppression mechanism. This helps to specifically solve the problems of data cleaning standardization bias and misleading entity association relationships caused by noisy data, sudden peak flows, and peak data. By suppressing data stream interference in a timely manner through the interference suppression mechanism, unqualified preprocessed data is prevented from entering the subsequent graph modeling process, ensuring the accuracy and reliability of preprocessed data in the subsequent processing, thereby providing a high-quality data foundation for multi-hop relationship retrieval in the data warehouse.

[0014] 3. During peak sales periods on e-commerce platforms, the high-load scenario of user behavior data processing results in an exponential increase in the scale of the data stream due to massive concurrent user operations. The data types are complex and diverse, and the data is further complicated by factors such as duplicate data caused by network fluctuations, fraudulent data generated by malicious order manipulation, and data transmission disorder caused by instantaneous traffic spikes. These factors combined can easily cause data stream interference, necessitating data stream interference assessment. The assessment result determines whether to trigger an interference suppression mechanism. This mechanism involves sending a prompt to designated personnel to re-collect and preprocess the data. Data collection and preprocessing involves collecting data from multiple sources and sequentially performing data cleaning, standardization, and noise reduction on the collected data. This helps to effectively identify data stream interference and address various types of interference within the data stream. The goal is to obtain qualified preprocessed data. Specifically, in high-load scenarios such as e-commerce platform promotions, it can effectively address the cascading effects of noisy data, sudden peak data streams, and third-level false association data. This avoids processing delays caused by noise redundancy occupying storage resources and information errors caused by data transmission congestion exacerbating association errors. The interference suppression mechanism is triggered by the data stream interference judgment result, allowing pre-defined personnel to process different levels of interference data according to priority. It prioritizes processing duplicate logs to release storage resources, diverts sudden data streams to alleviate transmission pressure, and verifies and corrects false associations to ensure data accuracy. This prevents the spread of interference and provides a reliable data source for subsequent graph modeling, multi-level graph index construction, and other stages. It helps reduce the problem of low accuracy of preprocessed data caused by data stream interference under high load conditions. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a fast data warehouse multi-hop relationship retrieval method based on graph embedding index provided in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the basic architecture of the fast data warehouse multi-hop relationship retrieval method based on graph embedding index provided in the embodiments of the present invention;

[0018] Figure 3 This is a logic diagram of the interference suppression mechanism of the fast data warehouse multi-hop relationship retrieval method based on graph embedding index provided in the embodiments of the present invention;

[0019] Figure 4 This is a schematic diagram of the structure of a fast data warehouse relationship retrieval system based on graph embedding index provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0021] like Figure 1 The diagram shown is a flowchart of a fast data warehouse relationship retrieval method based on graph embedding index provided in an embodiment of the present invention. Figure 1 It can be known that:

[0022] As an example of the first aspect, firstly, data flow interference monitoring: During the data warehouse data search process, the interference suppression mechanism is triggered based on the data flow interference judgment result. The interference suppression mechanism is used to improve the accuracy of overall data processing and analysis. After the data flow interference judgment is qualified, a graph model is obtained by modeling based on the obtained qualified preprocessed data. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization, and the preprocessed data represents the collected multi-source data. Data flow interference monitoring helps to filter noisy data and sudden peak flows in multi-source data flows, reduce the impact of data flow interference on data processing, and ensure that the preprocessed data on which the subsequent graph model is based, such as the association data between user identifiers and corresponding browsed product identifiers, and cross-purchase record data between different product categories, have high accuracy and reliability.

[0023] Secondly, index update timeliness monitoring: After graph modeling is completed, a multi-level graph index is built based on qualified preprocessed data and the graph model. The multi-level graph index is used to improve the ability of fast retrieval and accurate matching of multi-hop relationship queries in the data warehouse. The timeliness of index updates is judged based on the real-time changes of dynamic data flow. The index update timeliness judgment result determines whether to trigger the index step size update mechanism. The index step size update mechanism is used to improve the synchronization between the multi-level graph index and the real-time changes of dynamic data flow. By monitoring the timeliness of index updates, it is possible to ensure that the multi-level graph index can keep up with the changes of dynamic data flow in a timely manner, ensuring the timeliness and accuracy of the index, and providing strong support for efficient multi-hop relationship queries.

[0024] Next, multi-hop relationship query accuracy monitoring: After the index update timeliness is deemed satisfactory, multi-hop relationship queries are performed on the data warehouse based on the satisfactory multi-level graph index. The candidate node filtering range shrinking mechanism is triggered based on the multi-hop relationship query accuracy verification results. Triggering the candidate node filtering range shrinking mechanism is used to improve the matching degree between multi-hop relationship results and query targets. Multi-hop relationship query accuracy monitoring helps to improve the fit between multi-hop relationship query results and query targets, reduce invalid queries caused by result deviations, and improve the timeliness of complex multi-hop relationship queries.

[0025] Finally, data query response timeliness monitoring: After the accuracy of multi-hop relationship result query is verified, the timeliness of data query response is used to determine whether to trigger the query response optimization mechanism. The query response optimization mechanism is used to improve the real-time performance of data query processing. Data query response timeliness monitoring helps to speed up the response speed of multi-hop relationship queries in the data warehouse, reduce query latency, ensure that accurate query results can be returned quickly when facing dynamic data streams, and improve overall query efficiency.

[0026] It should be noted that a database storing various preset data was established before the design of the data warehouse multi-hop relationship fast retrieval method based on graph embedding index provided in this application. The database includes, but is not limited to, preset interference threshold, preset slight interference threshold, preset severe interference threshold, preset delay threshold, preset query qualification threshold, preset reliability threshold, preset response threshold, etc., among which various preset values ​​are directly set by technical personnel.

[0027] In this embodiment, the combined effects of data flow interference monitoring, index update timeliness monitoring, multi-hop relationship query accuracy monitoring, and data query response timeliness monitoring form a complete dynamic optimization closed loop. This ensures the accuracy of data processing and the timeliness of index updates during multi-hop relationship retrieval in the data warehouse, while also improving the accuracy of query results and the timeliness of responses. This effectively solves the problem of low efficiency in multi-hop relationship retrieval in the data warehouse caused by data flow interference. Specifically, data flow interference monitoring provides qualified preprocessed data and graph models for subsequent index update timeliness monitoring. If data flow interference is not effectively suppressed, leading to… Inaccurate preprocessed data will lead to biases in the multi-level graph index built upon it, affecting the timeliness and accuracy of index updates. The results of index update timeliness monitoring directly impact the accuracy monitoring of multi-hop relationship queries. If the multi-level graph index cannot keep up with the dynamic data flow, its timeliness and accuracy will be insufficient, resulting in a lower match between multi-hop relationship query results and the query target, increasing invalid queries. The results of multi-hop relationship query accuracy monitoring then affect the timeliness monitoring of data query response. If the multi-hop relationship query results have significant deviations, repeated query adjustments are required, prolonging query response time and affecting the timeliness of data query responses.

[0028] like Figure 2 The diagram shown illustrates the basic architecture of a fast data warehouse multi-hop relationship retrieval method based on graph embedding indexes, as provided in this embodiment of the invention. Figure 2It can be seen that during the data warehouse data search process, data flow interference is judged, and a data flow interference index is obtained. If the data flow interference index is less than a preset interference threshold, graph modeling is performed; otherwise, an interference suppression mechanism is triggered. After the interference suppression mechanism ends, if the data flow interference judgment still fails, an interference suppression failure alarm is sent; otherwise, graph modeling is performed. After graph modeling, multi-level graph indexes are constructed, and the timeliness of index updates is judged based on the real-time changes of dynamic data flow. An index update timeliness evaluation index is obtained, and if it is less than a preset delay threshold, a data warehouse multi-hop relationship query is performed; otherwise, an index step size update mechanism is triggered. After the index step size update mechanism ends, if the index update timeliness judgment still fails, a step size update failure alarm is sent; otherwise, the data warehouse... The system performs multi-hop relationship queries and obtains the query accuracy index for the multi-hop relationship results. It then determines whether the query accuracy index exceeds a preset acceptable threshold. If it does, a data query response timeliness assessment is performed. Otherwise, a candidate node filtering range narrowing mechanism is triggered. If the multi-hop relationship result query accuracy index is still unacceptable after the candidate node filtering range narrowing mechanism ends, an optimization failure alarm is sent. Otherwise, a data query response timeliness assessment is performed, and the data query response timeliness result is obtained. It then determines whether the data query response timeliness result is less than a preset response threshold. If it does, the data query response timeliness is deemed to meet the standard, and the current query processing strategy is maintained. Otherwise, a query response optimization mechanism is triggered. If the data query response timeliness assessment is still unacceptable after the query response optimization mechanism ends, a query response optimization failure alarm is sent. Otherwise, continuous monitoring of the data query response is performed.

[0029] Furthermore, the specific process for determining whether to trigger the interference suppression mechanism based on the data stream interference judgment result is as follows: Data stream interference data items are acquired to reflect the severity of various interference factors in the data stream and their impact on data quality. Data stream interference data items include the noise data verification value within the window, the burst peak stream verification value, and the peak data verification value. The noise data verification value is represented by the result after weighting the noise data impact weight coefficient by quantifying the noise data proportion within the preset interference detection time period and the preset noise data proportion. The preset interference detection time period represents the time period corresponding to the data stream interference judgment, and the proportion quantification represents the ratio calculation. The preset interference detection time period is monitored by a data quality monitoring instrument. The noise data proportion is calculated by quantifying the amount of redundant and repeated data and the total amount of data within a given time period. The pre-defined noise data proportion is represented by the average noise data proportion over historical time periods. The burst peak flow verification value is calculated by quantifying the duration of the burst peak flow within a pre-defined interference detection time period and the pre-defined burst peak flow duration, and then weighting it with a weighting coefficient based on the burst peak flow duration. Specifically, the burst peak flow duration is defined as the continuous duration for which the amount of pre-processed data collected within the pre-defined interference detection time period exceeds a pre-defined data volume threshold, which is pre-set by designated personnel. The preset duration of sudden peak flows is represented by the average duration of sudden peak flows over historical time periods. The peak data verification value is represented by the result of quantifying the proportion of peak data within the preset interference detection time period and the preset peak data proportion, and then weighting it with a peak data influence weighting coefficient. Specifically, the peak data proportion is the result of quantifying the proportion of preprocessed data collected within the preset interference detection time period that exceeds a preset data volume threshold, relative to the total data volume. The harmonic average of the data flow interference data items is used as the data flow interference index to reflect the degree to which preprocessed data is affected by noise and peak flows. When the duration of sudden peak flows... When the load increases, the data processing center's processing capacity is limited under prolonged high load, making it unable to effectively distribute and process the large influx of data in a timely manner. This leads to an increase in the proportion of peak data. As the accumulated peak data exceeds the processing threshold of the data verification mechanism, some data suffers from format corruption and information loss during transmission and storage, further increasing the proportion of data noise. The increased duration of the sudden peak flow, the proportion of peak data, and the proportion of data noise will further increase the data flow interference index. Data flow interference index qualification verification: Determine whether the data flow interference index is less than the preset interference threshold, where the preset interference threshold is represented by the average value of the data flow interference index over a historical time period.If so, the corresponding preprocessed data is marked as qualified preprocessed data, and a graph model is obtained based on the qualified preprocessed data and stored in the data warehouse. Otherwise, an interference suppression mechanism is triggered. Preprocessed data refers to data collected from multiple sources in a multi-source system. Data cleaning, structure mapping and type conversion, and data standardization are performed sequentially on the collected multi-source data. Data cleaning: Automated detection and cleaning of redundant values, noise, abnormal data, and formatted errors in the input data, such as removing meaningless blank fields and filling in missing values ​​from log data. Structure mapping and type conversion: Parsing and converting semi-structured data, such as XML (eXtensible Markup Language) files, into a form-like structure, and semantically aligning it with standard schema logic. Data standardization: Unifying units, time formats, encoding methods, etc., such as formatting timestamps to UTC (Coordinated Universal Time).

[0030] It should be noted that the weighting coefficients for noise data impact, duration of sudden peak flows impact, and peak data impact are used to reflect the degree of influence of data stream interference items on the data stream interference index. In this embodiment, there is a mapping group obtained from the database. This mapping group contains mapping sets, which are pre-set by professional technicians. The mapping relationships are in a one-to-one or many-to-one form, so as to accurately reflect the degree of influence of different interference factors on the data stream interference index under different data scenarios.

[0031] Specifically, a mapping relationship is established between the quantitative results of the proportion of data stream interference items and their corresponding weighted impact values. The weighted impact values ​​are represented by a 0-1 value range. The standard interference factor parameters include the preset noise data proportion rate, the preset duration of sudden peak flows, and the preset peak data proportion rate. The weighted impact values ​​include the noise data impact weight coefficient, the sudden peak flow duration impact weight coefficient, and the peak data impact weight coefficient. When the system receives the quantitative results of the proportion of data stream interference items, it can quickly retrieve the corresponding weighted impact values ​​from the pre-built mapping group, thereby accurately quantifying the degree of influence of data stream interference items on the data stream interference index. The noise data impact weight coefficient mainly affects the quantitative results related to the noise data proportion rate, the sudden peak flow duration impact weight coefficient mainly affects the quantitative results related to the duration of sudden peak flows, and the peak data impact weight coefficient affects the quantification of the peak data proportion rate. The three work together to form a comprehensive impact system on the data stream interference index, thereby improving the accuracy and adaptability of data stream interference judgment.

[0032] In this embodiment, by judging data stream interference, it is possible to comprehensively integrate key interference factors such as noise data, sudden peak streams, and peak data within the window. By quantitatively analyzing each interference data item and forming a data stream interference index, the degree of interference to the preprocessed data is accurately reflected. Combined with a preset interference threshold for qualification verification, it is possible to effectively determine whether an interference suppression mechanism needs to be triggered. This process not only achieves a scientific assessment of data stream interference but also enables targeted measures to reduce the impact of interference, ensuring that the preprocessed data used for subsequent graph modeling has high quality. This lays a reliable data foundation for the entire data warehouse multi-hop relationship retrieval process and improves the accuracy and stability of data processing and analysis.

[0033] like Figure 3 The diagram shown is a logic diagram of the interference suppression mechanism of the fast data warehouse multi-hop relationship retrieval method based on graph embedding index provided in an embodiment of the present invention. Figure 3 It can be seen that during the implementation of the interference suppression mechanism, the first step is to judge based on the data stream interference index: if the data stream interference index is not less than the preset interference threshold and is less than the preset slight interference threshold, the corresponding preprocessed data is marked as Level 1 noise data; if the data stream interference index is not less than the preset slight interference threshold and is less than the preset severe interference threshold, the corresponding preprocessed data is marked as Level 2 burst peak data; if the data stream interference index is not less than the preset severe interference threshold, the corresponding preprocessed data is marked as Level 3 false correlation data. Layered filtering is initiated for Level 1 noise data, dynamic buffering and diversion are implemented for Level 2 burst peak data, and rule verification and correction are performed for Level 3 false correlation data. After the interference suppression mechanism ends, the qualification verification of the interference suppression mechanism is performed. If the data stream interference index obtained again is still not less than the preset interference threshold, an interference suppression failure alarm is sent; otherwise, graph modeling is performed.

[0034] Furthermore, the specific process of the interference suppression mechanism is as follows: SS1, based on the data stream interference index, makes a judgment: if the data stream interference index is not less than a preset interference threshold and less than a preset slight interference threshold, then the corresponding preprocessed data is marked as Level 1 noise data; if the data stream interference index is not less than a preset slight interference threshold and less than a preset severe interference threshold, then the corresponding preprocessed data is marked as Level 2 burst peak stream data; if the data stream interference index is not less than a preset severe interference threshold, then the corresponding preprocessed data is marked as Level 3 spurious correlation data. The preset slight interference threshold and the preset severe interference threshold... All are pre-set by designated personnel; SS2, tiered filtering is initiated for Level 1 noise data: when Level 1 noise data is detected, a pre-set deduplication algorithm is used to identify and retain unique records, and duplicate pre-processed data is deleted. Tiered filtering for Level 1 noise data helps to efficiently remove redundant data. The pre-set deduplication algorithm, such as a hash deduplication algorithm, is pre-set by designated personnel; SS3, dynamic buffering and diversion are implemented for Level 2 burst peak data: when Level 2 burst peak data is detected, a temporary data buffer is activated to store excess data and import it into the processing flow in batches. Level 2 burst peak data is dynamically buffered and diverted. Buffering and offloading help ensure the stability of the data processing flow; SS4 performs rule verification and correction for Level 3 false association data: When Level 3 false association data is detected, based on a predefined business association rule library, each entity interaction record is verified, and false associations that do not conform to the rules are eliminated. That is, the database has a predefined business association rule library. For each detected Level 3 false association data, the corresponding rule in the rule library is automatically matched according to the type of the associated entity. If the corresponding rule matches successfully, the data is retained; otherwise, the record is eliminated. Specifically, Level 3 false association data... Row rule verification and correction help improve the authenticity of entity association; SS5, interference suppression mechanism qualification verification: After the interference suppression mechanism ends, the data flow interference index is reacquired. If the data flow interference index is still not less than the preset interference threshold, an interference suppression failure alarm is sent. Otherwise, the corresponding preprocessed data is marked as qualified preprocessed data, and a graph model is obtained based on the qualified preprocessed data and stored in the data warehouse. Among them, the qualified preprocessed data, core nodes, dynamic interaction nodes and semantic enhancement nodes in the historical data are randomly divided into training sets. The training set is input into the heterogeneous graph construction model fused by GNN (Graph Neural Network) for training to obtain the training model as the graph model. The reacquired qualified preprocessed data is input into the graph model, and the core nodes, dynamic interaction nodes and semantic enhancement nodes are output.

[0035] In this embodiment, the interference suppression mechanism helps reduce the negative impact of different levels of data stream interference on data processing. Since the impact of different levels of data stream interference on data quality varies, for example, Level 1 noise data mainly interferes with the redundancy of repeatedly reported data, with a relatively minor interference level; Level 2 burst peak data streams can disrupt the processing flow, with a slightly lower interference level; and Level 3 false association data can mislead entity associations, with the most severe interference level. Therefore, classifying the data stream interference data into three levels and implementing targeted optimization measures at each level can reduce resource waste or incomplete processing caused by single-processing. Combined with the qualification verification of the interference suppression mechanism, the impact of different levels of interference on preprocessed data can be minimized, thereby providing high-quality, qualified preprocessed data for graph model modeling.

[0036] Furthermore, a multi-level graph index is constructed based on qualified preprocessed data and a graph model, and the timeliness of index updates is judged based on real-time changes in dynamic data flow. The specific process is as follows: A multi-level graph index is constructed based on qualified preprocessed data and a graph model. The multi-level graph index means that nodes are first partitioned by type using an entity classification method and a first-level type index is created. Then, nodes and edges within the partition are mapped to low-dimensional vectors using a preset graph embedding algorithm. A second-level similarity index is constructed based on the low-dimensional vectors, and edge weight information is processed using a weighted hierarchical method to obtain a filter index. The filter indexes at each level are sequentially associated. The edge weight information reflects the tightness of the relationship between entity nodes. The preset graph embedding algorithm is set in advance by preset personnel, such as the Node2Vec graph embedding algorithm. After the multi-level graph index is constructed, it is associated with the graph model in the data warehouse, and the timeliness of index updates is judged based on real-time changes in dynamic data flow.

[0037] Specifically, the process for judging the timeliness of index updates is as follows: Obtain the index update timeliness evaluation index; after the multi-level graph index is built, monitor the time from when the newly acquired qualified preprocessed data is written to the data warehouse to when the multi-level graph index is updated using a time-series monitoring instrument. This timeliness evaluation index is used to reflect the response speed and synchronization efficiency of the multi-level graph index to changes in new data in the data warehouse; Verify the qualification of the index update timeliness evaluation index: determine whether the index update timeliness evaluation index is less than a preset delay threshold, where the preset delay threshold is represented by the average value of the index update timeliness evaluation index over a historical time period; if so, mark the corresponding multi-level graph index as a qualified multi-level graph index and perform a multi-hop relationship query in the data warehouse; otherwise, trigger the index step size update mechanism.

[0038] In this embodiment, by constructing a multi-level graph index, an efficient retrieval foundation is provided for multi-hop relationship queries in the data warehouse, which helps improve the speed and accuracy of queries. By judging the timeliness of index updates, it is helpful to promptly detect whether index updates are lagging, ensuring that the multi-level graph index can keep up with real-time data flow changes, avoiding query result deviations caused by index update lags, and improving the timeliness of multi-level graph index updates. The combination of these two aspects not only constructs a high-quality multi-level graph index, but also ensures the dynamic adaptability of the multi-level graph index, thus providing strong support for the timeliness of multi-hop relationship retrieval in the data warehouse.

[0039] Furthermore, the specific process of the index step size update mechanism is as follows: The index update timeliness evaluation index, the current index construction iteration step size, and the index association coverage are input into the database into an iteration step size mapping set used to dynamically optimize the index update rhythm. An iteration step size adjustment coefficient is then obtained. Specifically, when constructing a multi-level graph index, the interval length of each iteration update is monitored by an index iteration monitoring instrument and used as the current index construction iteration step size. The ratio of the actual number of entity associations included in the multi-level graph index to the total number of entity associations in the data warehouse is used as the index association coverage rate. The database contains a set of iteration step size mapping sets that reflect the correspondence between the combination of the index update timeliness evaluation index, the current index construction iteration step size, and the index association coverage rate, and the corresponding iteration step size adjustment coefficient. The iterative step size mapping set is pre-defined and stored in a database by designated personnel. For example, iterative step size mapping sets are gradually constructed through statistical analysis of historical index update data and iterative verification using pre-defined algorithms (such as gradient descent). First, specific combination parameters of index update timeliness evaluation indicators, the iterative step size of the current index construction, and the index association coverage rate are extracted from a large number of actual index update scenarios. Each parameter is assigned a weight based on the importance of index update efficiency. Simultaneously, the actual iterative step size adjustment coefficients for the corresponding scenarios are recorded. Then, abnormal correlation data is removed through correlation analysis (such as Spearman rank correlation coefficient analysis), retaining the statistically significant correspondence between the specific combination of index update timeliness evaluation indicators, the iterative step size of the current index construction, and the index association coverage rate, and the iterative step size adjustment coefficients.Iteration step size adjustment coefficient compliance check: This checks whether the iteration step size adjustment coefficient is within the preset reasonable step size threshold range. This preset reasonable step size threshold range is pre-set by designated personnel and includes both endpoints to reduce mismatches caused by a fixed single value. If the iteration step size adjustment coefficient exceeds the upper limit of the preset reasonable step size threshold range, it may cause excessive index jumps, leading to missed data updates or an unbalanced index structure, affecting data retrieval efficiency and accuracy. If the iteration step size adjustment coefficient is less than the lower limit of the preset reasonable step size threshold range, it may cause excessively frequent index adjustments, consuming system resources and reducing the overall performance of index updates. If both are within the acceptable range, the iteration step size adjustment coefficient is marked as a qualified iteration step size adjustment value; otherwise, an abnormal step size alarm is sent. Based on the qualified iteration step size adjustment value... The magnitude of the adjustment step size, gradually decreasing the index building iteration step size, helps ensure the effectiveness of index updates while reducing resource waste or index instability caused by large-scale adjustments at once, and gradually improves the response speed of index updates to dynamic data streams. Continuous monitoring of index update timeliness evaluation metrics is performed; execution stops when the index update timeliness evaluation metrics are found to be less than a preset delay threshold. The index building iteration step size is not less than a preset minimum step size threshold, which is pre-set by designated personnel. Index step size update mechanism qualification verification: After the index step size update mechanism ends, if the re-acquired index update timeliness evaluation metrics are still not less than the preset delay threshold, a step size update failure alarm is sent; otherwise, the corresponding multi-level graph index is marked as a qualified multi-level graph index, and a multi-hop relationship query of the data warehouse is performed.

[0040] In this embodiment, the index step size update mechanism helps to dynamically optimize the update rhythm of multi-level graph indexes, enabling the indexes to better follow the changes in dynamic data flow. The step-by-step iterative step size adjustment method can accurately adapt to changes in dynamic data flow, improve the timeliness and response speed of index updates, and avoid resource waste and index instability caused by over-adjustment or under-adjustment. At the same time, the effectiveness of the index step size update mechanism is guaranteed by the qualification verification, which ultimately improves the timeliness and reliability of multi-level graph index updates and provides efficient support for multi-hop relationship queries in data warehouses.

[0041] Furthermore, the specific process of multi-hop relationship query in the data warehouse is as follows: The query start node is input into the secondary similarity index mapping set, and the embedding vector in the secondary similarity index corresponding to that partition is obtained. Here, the query start node represents the initial entity object specified when the user initiates the query, and is the starting entity for the multi-hop relationship query. The database contains a set of secondary similarity index mapping sets to reflect the association between the query start node and the embedding vector in the secondary similarity index of the corresponding partition. The secondary similarity index mapping set is pre-set and stored in the database by designated personnel, for example, through statistical analysis of historical multi-hop relationship query data and iterative verification using preset algorithms (such as the K-nearest neighbor algorithm). Secondary similarity index mapping set: First, extract the query starting node and the corresponding embedding vector in the secondary similarity index from a large number of actual query scenarios. Simultaneously record the actual association between the query starting node and the embedding vector in the corresponding scenario. Then, remove abnormal association data through correlation analysis (such as cosine similarity analysis), and retain the statistically significant association between the query starting node and the embedding vector in the secondary similarity index. Based on the obtained preset hop count and preset retrieval algorithm, retrieve the candidate node vector with the highest similarity to the embedding vector of the starting node. The preset hop count and preset retrieval algorithm are set in advance by preset personnel, such as KD (K-Dimensional Tree) retrieval. After completing the expansion of the preset hop count, map the candidate path back to the original entity and relation data in the data warehouse, and finally return the multi-hop relation results that meet the conditions. Perform multi-hop relation result query accuracy verification. The original entity represents the entity unit in the data warehouse that carries specific business information but has not been processed by the preset graph embedding algorithm. The relation data represents the association information formed between the original entities in business interactions, which is used to reflect the connection between different original entities.

[0042] Specifically, the process for verifying the accuracy of multi-hop relationship query results is as follows: First, obtain the multi-hop relationship query accuracy index. This index is quantified by representing the proportion of successful multi-hop relationship paths matched using a preset matching algorithm within a preset verification sample set to the total number of query paths. The preset verification sample set and the preset matching algorithm are pre-set by a preset team, such as cosine similarity matching or edit distance matching. The multi-hop relationship query accuracy index reflects the degree of consistency between the query results and the actual business context, and determines the validity of the query results. Second, verify the validity of the multi-hop relationship query accuracy index by determining whether it exceeds a preset threshold. This threshold is represented by the average value of the multi-hop relationship query accuracy index over a historical time period. If it does, the multi-hop relationship result is marked as a valid multi-hop relationship result and stored in the query result cache. A timeliness assessment of the data query response is then performed. Conversely, if it does not exceed the threshold, a narrowing mechanism for the candidate node selection range is triggered.

[0043] In this embodiment, multi-hop relationship querying of data warehouse and accuracy verification of multi-hop relationship result query help to efficiently obtain multi-hop relationship results that meet the conditions, and ensure the consistency between the query results and the actual business, thereby improving the accuracy of the query results. Multi-hop relationship querying of data warehouse, with the help of secondary similarity index and preset retrieval algorithm, can quickly locate candidate nodes, complete multi-hop expansion and map back to the original data, ensuring query efficiency. Meanwhile, accuracy verification of multi-hop relationship result query, through quantitative indicators and qualification verification of multi-hop relationship result query accuracy indicators, can promptly identify inaccurate query results and trigger optimization mechanisms, further improving the accuracy of query results and laying a good foundation for subsequent judgment of the timeliness of data query response.

[0044] Furthermore, the specific process of the candidate node screening range narrowing mechanism is as follows: The multi-hop relationship result query accuracy index, the candidate node quantity threshold retrieved by the current preset retrieval algorithm, and the path relevance are input into a candidate node range mapping set used to improve the candidate node screening accuracy. The corresponding screening range narrowing ratio is then obtained. The candidate node quantity threshold retrieved by the current preset retrieval algorithm is pre-set by a pre-defined team. The path relevance, obtained by accumulating edge weight information, represents the tightness of the relationship between entity nodes on the candidate path in the multi-hop relationship query. The database contains a set of candidate node range mapping sets used to reflect the multi-hop relationship result query accuracy index and the current preset retrieval algorithm's retrieval... The relationship between the candidate node quantity threshold and path relevance combination and the corresponding shrinkage ratio of the screening range is established. The candidate node range mapping set is pre-set by designated personnel and stored in a database. For example, the candidate node range mapping set is gradually constructed through statistical analysis of historical multi-hop relationship query data and iterative verification of preset algorithms (such as random forest regression). First, the specific combination parameters of multi-hop relationship result query accuracy indicators, candidate node quantity threshold, and path relevance of preset retrieval algorithms are extracted from a large number of actual query scenarios, and each parameter is assigned a weight influence value based on the importance of the business scenario. The actual shrinkage ratio of the screening range under the corresponding scenario is recorded simultaneously. Then, correlation analysis (such as Pearson correlation) is used to further analyze the data. (Statistical analysis, etc.) Removes abnormally correlated data, retains statistically significant multi-hop relationship results, and establishes a specific combination of query precision indicators, candidate node number thresholds of the preset retrieval algorithm, and path correlation degree with the corresponding filtering range shrinkage ratio; Filtering range shrinkage ratio qualification check: Determines whether the filtering range shrinkage ratio is greater than the preset maximum filtering threshold, which is pre-set by preset personnel; if so, a filtering range shrinkage ratio failure alarm is sent; otherwise, the filtering range shrinkage ratio is marked as a qualified filtering range shrinkage value; Based on the magnitude corresponding to the qualified filtering range shrinkage value, the adjustment step size is used to gradually reduce the number of candidate nodes returned by the preset retrieval algorithm, which helps to ensure that the number of candidate nodes returned by the preset retrieval algorithm is not exceeded. Under the premise of omitting key candidate nodes, nodes with low relevance to the query target are gradually eliminated to improve the matching accuracy between candidate nodes and the query target. Gradual adjustment can accurately identify and retain nodes closely related to the query target, while gradually removing redundant nodes to avoid misjudgments that may be caused by one-time screening, reduce invalid calculations and redundant results, thereby improving the accuracy of multi-hop relationship query results; continuously monitor the query accuracy index of the re-acquired multi-hop relationship results, and stop execution when the multi-hop relationship result query accuracy index is detected to be greater than the preset query qualification threshold; the candidate node screening range narrowing mechanism is executed no more than the preset maximum number of times, which is set in advance by preset personnel;Verification of the candidate node selection range narrowing mechanism's compliance: After the candidate node selection range narrowing mechanism is completed, if the accuracy index of the multi-hop relationship result query is still not greater than the preset qualified query threshold, an optimization failure alarm is sent; otherwise, the multi-hop relationship result is marked as a qualified multi-hop relationship result, synchronously stored in the query result cache, and the timeliness of the data query response is judged.

[0045] In this embodiment, the candidate node filtering range narrowing mechanism helps improve the accuracy of multi-hop relationship query results, reduce invalid queries and redundant data. By adjusting the candidate node filtering range in a tiered manner, it can accurately retain nodes closely related to the query target while effectively eliminating irrelevant nodes, thereby improving the accuracy of multi-hop relationship query results. At the same time, the efficient operation of the mechanism is ensured through continuous monitoring and execution limit. Finally, the rationality of the candidate node filtering range narrowing adjustment is ensured through the qualification verification of the candidate node filtering range narrowing mechanism, providing reliable result support for the subsequent judgment of the timeliness of data query response.

[0046] Furthermore, the specific process for determining whether to trigger the query response optimization mechanism based on the data query response timeliness judgment result is as follows: Obtain the original data items of the data query response; the original data items of the data query response include the verification value of the average data query response time, the verification value of the longest data query response time, and the verification value of the data query timeout ratio; the verification value of the average data query response time is represented by the result after quantifying the proportion of the average data query response time within a preset query time period and the preset average response time, and then weighting it with the influence weight coefficient of the average data query response time. The preset query time period represents the time period for judging the timeliness of the data query response. All multi-hop relationship queries in the data warehouse from the client are monitored using data query response monitoring instruments. The time from initiating the query to returning the result is used as the average response time for data queries. The preset average response time is represented by the average of the average response times for data queries over a historical time period. The verification value for the longest response time for data queries is represented by the result after weighting the longest response time for data queries within a preset query time period and the preset longest response time, and then applying a weighting factor to the longest response time for data queries. Specifically, the maximum time taken to query a single piece of data, monitored by a single query timekeeping instrument, is used as the longest response time for data queries. The preset longest response time is represented by the average of the longest response times for data queries over a historical time period. The verification value for the data query timeout rate is determined by the ratio of the data query timeout rate within a preset query time period to the preset data query timeout rate. The query timeout rate is quantified and then weighted according to the weighting coefficients of the overall query timeout rate. Specifically, the query timeout rate is calculated by quantifying the number of queries whose response time exceeds a preset threshold to the total number of queries. The preset query timeout rate is represented by the average of historical query timeout rates over a given period. The timeliness of the query response is determined by harmonic averaging the original data items in the query response. As the average query response time increases, the overall processing load of the data processing center rises, leading to increased resource competition (such as increased CPU utilization and memory allocation delays). This results in a disproportionate allocation of processing resources for complex or high-priority queries. This leads to an increase in the longest response time for data queries. Since the probability of the longest response time exceeding the preset response threshold increases, and long-running queries consume system resources, causing delays in other query processing, the proportion of data query timeouts increases. Furthermore, an increase in the average response time, the longest response time, and the proportion of timeouts further exacerbates the issue of timeliness in data query responses. The timeliness of data query responses reflects the overall response speed and stability of multi-hop relational data queries. Verification of the timeliness of data query responses involves determining whether the timeliness of the data query response is less than the preset response threshold, which is represented by the average value of the timeliness of data query responses over a historical time period.If so, the data query response timeliness is deemed satisfactory, and the current query processing strategy is maintained. Otherwise, the query response optimization mechanism is triggered. The query processing strategy indicates that during multi-hop relationship queries in the data warehouse, when the data query response timeliness is satisfactory, the original query method based on multi-level graph indexes is maintained to execute the data query.

[0047] Specifically, the query response optimization mechanism operates as follows: It retrieves all associated paths and their corresponding path confidence scores from the qualified multi-hop relationship results in the data warehouse. It then determines whether the path confidence score is less than a preset confidence threshold, which is represented by the average path confidence score over a historical time period. If so, the corresponding associated path is marked as a redundant path and directly deleted; otherwise, the corresponding associated path is retained as a valid associated path. Query response optimization mechanism qualification verification: After the query response optimization mechanism completes, the data query response timeliness result is retrieved again. If the data query response timeliness result is still not less than the preset response threshold, a query response optimization failure alarm is sent; otherwise, continuous monitoring of the data query response is performed.

[0048] It is important to note that the weighting coefficients for the average response time of data queries, the longest response time of data queries, and the proportion of data query timeouts are used to reflect the degree of influence of the original data items in the data query response on the timeliness of the data query response. In this embodiment, there is a mapping group obtained from the database. This mapping group contains a mapping set, which is pre-set by professional technicians. The mapping relationship in this group adopts a one-to-one correspondence or many-to-one form, so as to accurately reflect the degree of influence of different data query response factors on the timeliness of the data query response in different query scenarios.

[0049] Specifically, a mapping relationship is established between the quantified proportion of the original data items in the data query response and their corresponding weight influence values. The weight influence value proportion is represented by a 0-1 value range. Standard query response parameters include a preset average response time, a preset maximum response time, and a preset data query timeout rate. Weight influence values ​​include the weight coefficients for the average data query response time, the longest data query response time, and the data query timeout rate. When the system receives the quantified proportion of the original data items in the data query response, it can quickly retrieve the corresponding weights from the pre-built mapping group. The impact value is used to accurately quantify the degree of influence of the original data items on the timeliness of the data query response. Among them, the average response time of the data query mainly affects the quantitative results related to the average response time of the data query, the longest response time of the data query mainly affects the quantitative results related to the longest response time of the data query, and the timeout rate of the data query affects the quantification of the timeout rate of the data query. The three factors work together to form a comprehensive impact system on the timeliness of the data query response, thereby improving the accuracy and adaptability of the judgment on the timeliness of the data query response.

[0050] In this embodiment, the query response timeliness judgment and query response optimization mechanism help improve the efficiency of multi-hop relationship queries in the data warehouse. The query response timeliness judgment, through multi-dimensional consideration of the original data items in the query response, can objectively reflect the overall performance of multi-hop relationship queries and provide reasonable quantitative indicators for triggering the query response optimization mechanism. The query response optimization mechanism, by eliminating redundant association paths with low confidence, effectively reduces invalid data processing and significantly improves query response speed and stability. Through the effective combination of query response timeliness judgment and query response optimization mechanism, the stability of the query processing strategy is ensured, and the query response optimization mechanism can be intervened in a timely manner when the query response timeliness judgment is unqualified, thereby improving the efficiency of multi-hop relationship queries in the data warehouse.

[0051] like Figure 4The diagram shown is a structural schematic of a fast multi-hop relationship retrieval system for data warehouses based on graph embedding indexes, provided in an embodiment of this application. This system applies a fast multi-hop relationship retrieval method for data warehouses based on graph embedding indexes. Its features include: a data flow interference monitoring module, an index update timeliness monitoring module, a multi-hop relationship query accuracy monitoring module, and a data query response timeliness monitoring module. Specifically, the data flow interference monitoring module determines whether to trigger an interference suppression mechanism based on the data flow interference judgment result. After the data flow interference judgment is qualified, it performs graph modeling based on the acquired qualified preprocessed data to obtain a graph model. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization. The preprocessed data represents the collected multi-source data; the index update timeliness monitoring module is used to construct a multi-level graph index based on qualified preprocessed data and the graph model after the graph model modeling is completed, and to judge the index update timeliness based on the real-time changes of dynamic data flow, and to determine whether to trigger the index step size update mechanism based on the index update timeliness judgment result; the multi-hop relationship query accuracy monitoring module is used to perform multi-hop relationship queries on the data warehouse based on qualified multi-level graph index after the index update timeliness judgment is qualified, and to determine whether to trigger the candidate node screening range shrinking mechanism based on the multi-hop relationship result query accuracy verification result; the data query response timeliness monitoring module is used to determine whether to trigger the query response optimization mechanism based on the data query response timeliness judgment result after the multi-hop relationship result query accuracy verification is qualified.

[0052] In this embodiment, the data flow interference monitoring module, the index update timeliness monitoring module, the multi-hop relationship query accuracy monitoring module, and the data query response timeliness monitoring module exhibit a closely collaborative, progressive, and mutually supportive relationship. The data flow interference monitoring module obtains qualified preprocessed data through data cleaning and standardization and data flow interference suppression, providing high-quality qualified preprocessed data for subsequent graph model building. Based on the data flow interference monitoring module, the index update timeliness monitoring module constructs and dynamically optimizes a multi-level graph index to ensure that the multi-level graph index keeps up with data changes, providing an efficient retrieval foundation for multi-hop relationship queries. Its judgment result on the index update timeliness directly affects whether the multi-hop relationship query accuracy monitoring module can base its work on the qualified multi-level graph index. The system initiates query operations. The multi-hop relationship query accuracy monitoring module executes queries based on qualified multi-level graph indexes and ensures the accuracy of query results through multi-hop relationship result query accuracy verification. When the multi-hop relationship result query accuracy verification fails, a candidate node filtering range narrowing mechanism is triggered. The qualified multi-hop relationship results output by this module serve as the basis for the data query response timeliness monitoring module's response judgment. Based on the qualified multi-hop relationship result query accuracy verification, the data query response timeliness monitoring module further optimizes the query response speed to ensure that the system can return results quickly and stably. These four modules are interconnected, forming a complete closed-loop system that jointly ensures the efficiency, accuracy, and timeliness of multi-hop relationship fast retrieval in the data warehouse based on graph embedded indexes.

[0053] As an embodiment of the second aspect, during the data warehouse data search process, data flow interference is judged, and a data flow interference index is obtained. It is determined whether the data flow interference index is less than a preset interference threshold. If so, graph modeling is performed; otherwise, an interference suppression mechanism is triggered. This mechanism sends a prompt to a preset group of personnel to re-collect and preprocess the data. After the interference suppression mechanism ends, if the data flow interference judgment is still unsatisfactory, an interference suppression failure alarm is sent. Otherwise, graph modeling is performed. After graph modeling, a multi-level graph index is constructed, and the timeliness of index updates is judged based on the real-time changes of the dynamic data flow. An index update timeliness evaluation index is obtained, and it is determined whether the index update timeliness evaluation index is less than a preset delay threshold. If so, a multi-hop relationship query in the data warehouse is performed; otherwise, an index step size update mechanism is triggered. After the index step size update mechanism ends, if the index update timeliness judgment is still unsatisfactory, an interference suppression failure alarm is sent. If the step size update fails, an alarm is sent. Otherwise, a multi-hop relationship query is performed in the data warehouse, and the accuracy index of the multi-hop relationship query result is obtained. It is determined whether the accuracy index of the multi-hop relationship query result is greater than the preset query qualification threshold. If it is, a data query response timeliness judgment is performed. Otherwise, a candidate node screening range narrowing mechanism is triggered. After the candidate node screening range narrowing mechanism ends, if the multi-hop relationship query accuracy index is still unqualified, an optimization failure alarm is sent. Otherwise, a data query response timeliness judgment is performed, and the data query response timeliness result is obtained. It is determined whether the data query response timeliness result is less than the preset response threshold. If it is, the data query response timeliness is deemed to meet the standard, and the current query processing strategy is maintained. Otherwise, a query response optimization mechanism is triggered. After the query response optimization mechanism ends, if the data query response timeliness judgment is still unqualified, a query response optimization failure alarm is sent. Otherwise, continuous monitoring of data query response is performed.

[0054] Based on the first aspect of the embodiment, in the high-load scenario of user behavior data processing during e-commerce platform promotions, the platform not only has to deal with the explosive growth of user browsing and ordering behavior data in a short period of time, but also needs to process data such as product information updates and inventory change records simultaneously. The server computing power and data transmission channels are both under high load: a large number of repeated click logs occupy storage resources, causing processing delays in the sudden order data stream; and the information disorder caused by data transmission congestion will further exacerbate the generation of false correlation data, forming a chain problem from noise redundancy to processing delays, and then to correlation errors.

[0055] In this embodiment, during high-load scenarios of user behavior data processing on e-commerce platforms during major promotional periods, such as a surge in user visits, order requests, and product search frequency, data flow interference is assessed and an interference index is obtained. If the interference index exceeds a preset interference threshold, triggering an interference suppression mechanism, the e-commerce data management platform will execute intervention measures in priority order. First, for noisy data, pre-defined personnel are prompted to prioritize processing duplicate user click logs, and redundant data is stored separately through temporary expansion of storage partitions to avoid occupying core data processing channels. Next, for sudden peak data flows, order data flows for popular products are traffic-diverted, and processing queues are allocated according to user levels to reduce the impact of high concurrency on data transmission channels. Finally, for false association data, personnel are guided to perform batch verification of incorrectly associated shipping addresses and payment accounts, and a predefined business association rule base is used to quickly correct mapping relationships, ensuring that the accuracy of key transaction data is prioritized. This helps to promptly block the chain reaction of data flow interference and ensure the stability and reliability of data processing on the e-commerce platform during major promotional periods.

[0056] In summary, this application embodiment determines whether to trigger an interference suppression mechanism based on the data flow interference judgment result. This interference suppression mechanism improves the overall accuracy of data processing and analysis. After the data flow interference judgment is deemed satisfactory, a graph model is created based on the acquired satisfactory preprocessed data. This helps reduce the interference of noisy data and sudden peak flows on data cleaning, standardization, and entity relationships, laying a reliable data foundation for subsequent processing. The satisfactory preprocessed data refers to the preprocessed data obtained after cleaning and standardization, representing multi-source data collected. After graph modeling is completed, a multi-level graph index is constructed based on the satisfactory preprocessed data and the graph model. This multi-level graph index enhances the ability to quickly retrieve and accurately match multi-hop relationship queries in the data warehouse. Furthermore, the timeliness of index updates is judged based on real-time changes in the dynamic data flow. The timeliness judgment result determines whether to trigger an index step size update mechanism, which helps improve the timeliness and accuracy of graph index construction, ensuring its adaptability to dynamic data flow changes. The index step size update mechanism is used to improve the real-time synchronization of multi-level graph indexes with dynamic data streams. After the index update timeliness judgment is qualified, multi-hop relationship queries are performed on the data warehouse based on the qualified multi-level graph index. The candidate node filtering range shrinking mechanism is triggered based on the query accuracy verification result of the multi-hop relationship results. This helps to enhance the ability of complex multi-hop relationship queries to cope with dynamic data streams and non-linear retrieval needs, improve path traversal efficiency, and trigger the candidate node filtering range shrinking mechanism to improve the matching degree between multi-hop relationship results and query targets. After the multi-hop relationship result query accuracy verification is qualified, the query response optimization mechanism is triggered based on the data query response timeliness judgment result. This helps to reduce data query response latency and improve the multi-hop relationship retrieval efficiency of the data warehouse. The query response optimization mechanism is triggered to improve the real-time performance of data query processing. This achieves the improvement of multi-hop relationship retrieval efficiency of the data warehouse and effectively solves the problem of low multi-hop relationship retrieval efficiency of the data warehouse due to data stream interference in the existing technology.

[0057] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0058] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, systems, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

Claims

1. A fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding indexes, characterized in that, include: During the data warehouse data search process, the interference suppression mechanism is triggered based on the data flow interference judgment result. The interference suppression mechanism is used to improve the accuracy of overall data processing and analysis. After the data flow interference judgment is qualified, the graph model is obtained by graph modeling based on the qualified preprocessed data. The qualified preprocessed data refers to the preprocessed data obtained after cleaning and standardization. The preprocessed data refers to the collected multi-source data. After the graph model is completed, a multi-level graph index is constructed based on qualified preprocessed data and the graph model. The timeliness of index updates is judged based on the real-time changes of dynamic data flow. The index update timeliness judgment result determines whether to trigger the index step size update mechanism. The multi-level graph index is used to improve the ability of fast retrieval and accurate matching of multi-hop relationship queries in the data warehouse. The index step size update mechanism is used to improve the synchronization between the multi-level graph index and the real-time changes of dynamic data flow. After the index update timeliness is deemed satisfactory, a multi-hop relationship query of the data warehouse is performed based on the satisfactory multi-level graph index. The candidate node filtering range shrinking mechanism is then triggered based on the query accuracy verification result of the multi-hop relationship result. The triggering of the candidate node filtering range shrinking mechanism is used to improve the matching degree between the multi-hop relationship result and the query target. After the accuracy of the multi-hop relationship query result is verified, a decision is made on whether to trigger the query response optimization mechanism based on the data query response timeliness judgment result. The triggering of the query response optimization mechanism is used to improve the real-time performance of data query processing.

2. The fast data warehouse multi-hop relationship retrieval method based on graph embedding index according to claim 1, characterized in that, The process of determining whether to trigger the interference suppression mechanism based on the data stream interference judgment result is as follows: Data stream interference data items are obtained to reflect the severity of various interference factors in the data stream and their impact on data quality. The data stream interference data items include the window noise data verification value, the burst peak stream verification value, and the peak data verification value. The noise data verification value is represented by the result of quantifying the proportion of noise data within a preset interference detection time period and the preset noise data proportion, and then weighting it with the noise data influence weighting coefficient. The noise data proportion is represented by the result of quantifying the proportion of redundant information duplicate data within the preset interference detection time period and the total data volume. The burst peak flow verification value is represented by the result of quantifying the proportion of the burst peak flow duration within the preset interference detection time period and the preset burst peak flow duration, and then weighting it with the burst peak flow duration influence weighting coefficient. The burst peak flow duration is represented by the continuous duration when the amount of data corresponding to the preprocessed data collected within the preset interference detection time period exceeds the preset data amount threshold. The peak data verification value is represented by the result of quantifying the proportion of peak data within a preset interference detection time period and the preset peak data proportion, and then weighting it with the peak data influence weight coefficient. The peak data proportion is represented by the result of quantifying the proportion of data corresponding to the preprocessed data collected within the preset interference detection time period that exceeds the preset data volume threshold and the total data volume. The result of harmonic averaging of the data stream interference items is used as the data stream interference index to reflect the degree to which the preprocessed data is affected by noise and peak flow. If the interference index of the data stream is less than the preset interference threshold, the corresponding preprocessed data is marked as qualified preprocessed data, and a graph model is obtained based on the qualified preprocessed data and stored in the data warehouse. Otherwise, the interference suppression mechanism is triggered.

3. The fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding index as described in claim 2, characterized in that, The interference suppression mechanism refers to sending a prompt to a designated person to re-collect and preprocess the data.

4. The fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding index according to claim 2, characterized in that, The specific process of the interference suppression mechanism is as follows: SS1, based on the data stream interference index, makes the following judgments: if the data stream interference index is not less than the preset interference threshold and is less than the preset slight interference threshold, the corresponding preprocessed data is marked as Level 1 noise data; if the data stream interference index is not less than the preset slight interference threshold and is less than the preset severe interference threshold, the corresponding preprocessed data is marked as Level 2 burst peak stream data; if the data stream interference index is not less than the preset severe interference threshold, the corresponding preprocessed data is marked as Level 3 false correlation data. SS2, when detecting Level 1 noise data, identifies and retains a unique record through a preset deduplication algorithm, and deletes duplicate pre-processed data; SS3: When a secondary burst peak flow data is detected, a temporary data buffer is enabled to store the excess data and import it into the processing flow in batches. SS4, when detecting false association data at level 3, verifies entity interaction records one by one based on a predefined business association rule base and removes false associations that do not conform to the rules; SS5, Interference Suppression Mechanism Qualification Verification: After the interference suppression mechanism ends, the data stream interference index is reacquired. If the data stream interference index is still not less than the preset interference threshold, an interference suppression failure alarm is sent. Otherwise, the corresponding preprocessed data is marked as qualified preprocessed data, and a graph model is obtained based on the qualified preprocessed data and stored in the data warehouse.

5. The fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding index according to claim 1, characterized in that, The process of constructing a multi-level graph index based on qualified preprocessed data and a graph model, and judging the timeliness of index updates based on real-time changes in dynamic data streams, is as follows: A multi-level graph index is constructed based on qualified preprocessed data and graph models. The multi-level graph index means that the nodes are first partitioned by type through entity classification method and a first-level type index is created. Then, the nodes and edges in the partition are mapped to low-dimensional vectors through a preset graph embedding algorithm. A second-level similarity index is constructed based on the low-dimensional vectors and the edge weight information is processed through a weight hierarchical method to obtain a filter index. The filter indexes at each level are then associated sequentially. After the multi-level graph index is constructed, it is associated with the graph model in the data warehouse, and the timeliness of index updates is judged based on the real-time changes of dynamic data flow. Obtain an index update timeliness evaluation metric. The index update timeliness evaluation metric is a representation of the time from when qualified preprocessed data is re-acquired and written into the data warehouse after the multi-level graph index is built to when the multi-level graph index is updated. The index update timeliness evaluation metric is used to reflect the response speed and synchronization efficiency of the multi-level graph index to new data changes in the data warehouse. If the index update timeliness assessment metric is less than the preset delay threshold, the corresponding multi-level graph index is marked as a qualified multi-level graph index, and a multi-hop relationship query of the data warehouse is performed; otherwise, the index step size update mechanism is triggered.

6. The fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding index according to claim 5, characterized in that, The index step size update mechanism works as follows: Input the index update timeliness evaluation metrics, the iteration step size of the current index construction, and the index association coverage into the iteration step size mapping set in the database for dynamically optimizing the index update rhythm, and obtain the iteration step size adjustment coefficient. Determine whether the iteration step size adjustment coefficient is within the preset reasonable step size threshold range. If it is, mark the iteration step size adjustment coefficient as a qualified iteration step size adjustment value; otherwise, send a step size abnormality alarm. The adjustment step size is determined by the magnitude corresponding to the qualified iteration step size adjustment value. The index construction iteration step size is gradually reduced, and the index update timeliness evaluation index is continuously monitored. When the index update timeliness evaluation index is found to be less than the preset delay threshold, the execution is stopped. After the index step update mechanism ends, if the timeliness evaluation index of the re-acquired index update is still not less than the preset delay threshold, an alarm for step update failure will be sent; otherwise, the corresponding multi-level graph index will be marked as a qualified multi-level graph index, and a multi-hop relationship query of the data warehouse will be performed.

7. The fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding index as described in claim 6, characterized in that, The specific process of the multi-hop relationship query in the data warehouse is as follows: Input the query start node into the secondary similarity index mapping set, obtain the embedding vector in the secondary similarity index corresponding to the partition, and retrieve the candidate node vector with the highest similarity to the embedding vector of the start node based on the obtained preset hop count and preset retrieval algorithm; After completing the expansion of the preset number of hops, the candidate paths are mapped back to the original entities and relational data in the data warehouse, and finally the multi-hop relational results that meet the conditions are returned. The accuracy of the multi-hop relational result query is verified. The original entities refer to the entity units in the data warehouse that carry specific business information but have not been processed by the preset graph embedding algorithm. The relational data refers to the association information formed between the original entities in business interactions, which is used to reflect the connection between different original entities. The multi-hop relationship result query accuracy verification means: obtaining a multi-hop relationship result query accuracy index. The multi-hop relationship result query accuracy index is quantified by the proportion of the number of multi-hop relationship paths successfully matched based on a preset matching algorithm in the preset verification sample set and the total number of query paths. The multi-hop relationship result query accuracy index is used to reflect the degree of consistency between the multi-hop relationship query results and the actual business relationship and to determine whether the query results are valid. If the accuracy index of the multi-hop relationship result query is greater than the preset qualified query threshold, the multi-hop relationship result is marked as a qualified multi-hop relationship result and stored in the query result cache library. The timeliness of data query response is also judged. Otherwise, the candidate node screening range narrowing mechanism is triggered.

8. The fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding index according to claim 7, characterized in that, The candidate node screening range narrowing mechanism works as follows: Input the multi-hop relationship result query accuracy index, the threshold for the number of candidate nodes retrieved by the current preset retrieval algorithm, and the path relevance into the candidate node range mapping set used to improve the candidate node filtering accuracy, and obtain the corresponding filtering range shrinkage ratio; Determine whether the filter range shrinkage ratio is greater than the preset maximum filter threshold. If so, send a filter range shrinkage ratio failure alarm; otherwise, mark the filter range shrinkage ratio as a qualified filter range shrinkage value. The adjustment step size is based on the magnitude corresponding to the shrinkage value of the qualified screening range. The number of candidate nodes returned by the preset retrieval algorithm is reduced step by step. The accuracy index of the re-acquired multi-hop relationship results is continuously monitored. When the accuracy index of the multi-hop relationship results is detected to be greater than the preset qualified threshold, the execution is stopped. After the candidate node screening range narrowing mechanism ends, if the multi-hop relationship result query accuracy index is still not greater than the preset query qualification threshold, an optimization failure alarm will be sent. Otherwise, the multi-hop relationship result will be marked as a qualified multi-hop relationship result, synchronously stored in the query result cache library, and the timeliness of data query response will be judged.

9. The fast retrieval method for multi-hop relationships in a data warehouse based on graph embedding index according to claim 1, characterized in that, The process of determining whether to trigger the query response optimization mechanism based on the timeliness of the data query response is as follows: Obtain the raw data items of the data query response, including the verification value of the average response time of the data query, the verification value of the longest response time of the data query, and the verification value of the proportion of data query timeouts; The average response time verification value for data queries is represented by the result of quantifying the average response time for data queries within a preset query time period and the preset average response time as a proportion, and then weighting it with the weighting coefficient of the average response time for data queries. The average response time for data queries is represented by quantifying the total time from the client initiating all multi-hop relationship queries of the data warehouse to the return of the query result as a proportion of the total amount of data of the target data being queried. The data query longest response time verification value is represented by the result of quantifying the proportion of the data query longest response time within a preset query time period and the preset longest response time, and then weighting it with the influence weight coefficient of the data query longest response time. The data query longest response time is represented by the maximum time spent querying a single piece of query data. The data query timeout ratio verification value is represented by the result of quantifying the proportion of data query timeout ratio within a preset query time period and the preset data query timeout ratio, and then weighting it with the influence weight coefficient of the data query timeout ratio. The data query timeout ratio is represented by quantifying the proportion of the number of query data whose query data response time exceeds a preset response threshold to the total number of query data. The result of harmonic averaging of the original data items in the data query response is used as the timeliness result of the data query response, which reflects the overall response speed and stability of multi-hop relationship data queries. Determine if the timeliness of the data query response is less than the preset response threshold. If so, determine that the timeliness of the data query response meets the standard and maintain the current query processing strategy; otherwise, trigger the query response optimization mechanism. The specific process of the query response optimization mechanism is as follows: retrieve all associated paths and their corresponding path confidence from the qualified multi-hop relationship results in the data warehouse, determine whether the path confidence is less than the preset confidence threshold, if so, mark the corresponding associated path as a redundant path and delete the redundant path directly, otherwise, retain the corresponding associated path as a valid associated path. After the query response optimization mechanism ends, the data query response timeliness result is retrieved again. If the data query response timeliness result is still not less than the preset response threshold, a query response optimization failure alarm is sent; otherwise, continuous monitoring of the data query response is performed.

10. A fast data warehouse multi-hop relationship retrieval system based on graph embedding index, employing the fast data warehouse multi-hop relationship retrieval method based on graph embedding index as described in any one of claims 1-9, characterized in that, include: Data flow interference monitoring module, index update timeliness monitoring module, multi-hop relationship query accuracy monitoring module, and data query response timeliness monitoring module: The data stream interference monitoring module is used to determine whether to trigger the interference suppression mechanism based on the data stream interference judgment result. After the data stream interference judgment is qualified, the module performs graph modeling based on the qualified preprocessed data to obtain the graph model. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization. The preprocessed data represents the collected multi-source data. The index update timeliness monitoring module is used to construct a multi-level graph index based on qualified preprocessed data and the graph model after the graph model modeling is completed, and to judge the index update timeliness based on the real-time changes of dynamic data flow. Based on the index update timeliness judgment result, it determines whether to trigger the index step size update mechanism. The multi-hop relationship query accuracy monitoring module is used to perform multi-hop relationship queries on the data warehouse based on the qualified multi-level graph index after the index update timeliness judgment is qualified, and to determine whether to trigger the candidate node screening range shrinkage mechanism based on the multi-hop relationship query accuracy verification result. The data query response timeliness monitoring module is used to determine whether to trigger the query response optimization mechanism based on the data query response timeliness judgment result after the multi-hop relationship result query accuracy verification is qualified.

Images