Quick retrieval method and system for multi-hop relationship of data warehouse based on graph embedded index

Abstract

The invention discloses a data warehouse multi-hop relationship quick retrieval method and system based on a graph embedded index, and relates to the technical field of multi-hop relationship retrieval data processing. The quick retrieval method for the multi-hop relationship of the data warehouse based on the graph embedded index comprises the following steps of: monitoring data flow interference; monitoring index updating timeliness; monitoring the query accuracy of the multi-hop relationship; and timely monitoring of data query response. According to the method, whether to perform graph model modeling to obtain the graph model is determined by judging whether to trigger the interference suppression mechanism, then the multi-level index is constructed based on the graph model, and whether to trigger the index step length updating mechanism is determined based on the index updating timeliness judgment result; and finally, based on a multi-hop relationship result query accuracy verification result, determining whether to trigger a candidate node screening range shrinkage mechanism, thereby achieving the effect of improving the multi-hop relationship retrieval efficiency of the data warehouse, and solving the problem of low multi-hop relationship retrieval efficiency of the data warehouse caused by data flow interference in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-hop relationship retrieval data processing, and in particular to a method and system for fast retrieval of multi-hop relationships in a data warehouse based on graph embedding indexes. Background Art

[0002] In the process of data retrieval based on graph embedding index, firstly, data collection tools (such as Kafka, Flume, etc.) are used to capture business (such as social network interaction) source data in real time, including streaming log data, structured form data and unstructured file data, and the collected data (such as user click stream logs, etc.) are stored in the data warehouse; then data analysis techniques (such as static analysis, etc.) are used to convert the table structure, foreign key relationship and dimension association in the data warehouse into a directed heterogeneous graph, and then graph embedding algorithms (such as Node2Vec, DeepWalk, etc.) are used to generate low-dimensional embedding vectors of nodes and edges, and combined with indexing strategies such as HNSW (Hierarchical Navigable Small World) to build a hierarchical index structure; when the user submits a multi-hop query, the data query center first uses a parser such as SQL (Structured Query It uses a Structured Query Language (SQL) parser to convert query statements into graph path expressions, decomposes complex queries into atomic subqueries, and generates corresponding embedding vector query conditions. It also uses an active node marking mechanism to iteratively expand neighbor nodes, combined with subgraph partitioning technology to reduce cross-subgraph computation. Distributed storage and query systems, such as ElasticSearch and HDFS (Hadoop Distributed File System), are used for data storage and query. Spark aggregates the results and converts them into structured data for return. Finally, when the data warehouse changes, retrieval technologies (such as MerkleTree) are used to detect the changes. Dynamic metadata labeling technology is used to regenerate embedding vectors and update indexes only for the changed nodes to maintain consistency.

[0003] In the process of data retrieval in a data warehouse, existing technologies first analyze the database table type, column name, and inter-table constraint relationships to distinguish between data tables that store specific objects and relational tables that store relationships; then, each data table is mapped to a concept node, each record in the table generates a corresponding entity node, and an "include" relationship is established from the concept node to the entity node; then, inter-entity relationships are constructed based on foreign key constraints or relational tables; the knowledge graph is then stored in a graph database (such as Google Cayley) and a Gremlin syntax query interface is provided; when a user enters a natural language query, the data query center first converts the query into a graph path expression through semantic parsing, then uses Gremlin to perform multi-hop relationship retrieval path matching entities in the knowledge graph, and finally returns the search results in the form of structured data.

[0004] For example, the Chinese invention patent announcement with announcement number: CN116108238B discloses a method, system, and device for optimizing multi-hop queries in a graph database, including: first, parsing the multi-hop query statement in the graph database, extracting key information such as the starting node and relationship type in the query, and generating an initial query plan based on the metadata of the graph database to determine the candidate query path direction; then, optimizing the initial query plan based on the distribution characteristics of nodes and relationships in the graph database, and screening out efficient query paths by calculating the selectivity estimates of different paths; then, during the execution of the multi-hop query, dynamically monitoring the query progress and the size of the intermediate results. When it is detected that the intermediate result set is too large, a pruning strategy is triggered to remove candidate nodes or paths with low relevance; finally, the results returned by the multi-hop query are checked for consistency and deduplication to ensure that the results meet the query semantics and accuracy requirements, and the optimized query plan is cached in the data warehouse.

[0005] For example, the Chinese invention patent announcement with announcement number: CN117556086B discloses a multi-hop path query method, device, computer equipment and storage medium, which include: first processing the original data, constructing a graph data structure including nodes, edges and edge attributes, and establishing an auxiliary index; then receiving a query request including parameters such as the starting node, the end node, the maximum number of hops, and the edge attribute constraints, and initializing the search queue, the visited node set and the hop counter based on the starting node; then iteratively expanding the path, traversing the neighbors of the current node that meet the constraints to generate a new path, and applying a pruning strategy to remove invalid paths that exceed the hop limit, contain loops or cannot reach the end point, and giving priority to expanding paths that are more likely to reach the end point; then performing constraint verification and deduplication screening on the searched end point path; finally, sorting the qualified paths according to the rules and returning the results.

[0006] It is found that the above technology has at least the following technical problems: During the data warehouse data search process, noise data in multi-source data streams (such as repeatedly reported redundant information and abnormal data with disordered formats) and sudden peak flows (such as intensive alarm data during equipment failures) may cause data flow interference, resulting in deviations in data cleaning and standardization processing, and entity association relationships being misled by false interactions. When updating the graph index based on dynamic data, the timeliness and accuracy of graph index construction are significantly reduced. It is difficult to cope with dynamic data streams and nonlinear retrieval requirements in complex multi-hop relationship queries, resulting in low path traversal efficiency and query response delays. There is a problem of low efficiency in multi-hop relationship retrieval in data warehouses due to data flow interference. Summary of the Invention

[0007] To address the problem of low efficiency in retrieval of multi-hop relationships in data warehouses due to data flow interference in the prior art, embodiments of the present invention provide a method and system for fast retrieval of multi-hop relationships in data warehouses based on graph embedding indexes. The technical solution is as follows: On the one hand, a method for rapid retrieval of multi-hop relationships in a data warehouse based on a graph embedded index is provided, which includes: during a data warehouse data search process, determining whether to trigger an interference suppression mechanism based on a data flow interference judgment result; after the data flow interference judgment is qualified, performing graph model building based on the obtained qualified preprocessed data to obtain a graph model, wherein the qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization, and the preprocessed data represents the collected multi-source data; after the graph model building is completed, constructing a multi-level graph index based on the qualified preprocessed data and the graph model, and performing index update timeliness judgment based on the real-time changes of the dynamic data flow, and determining whether to trigger an index step update mechanism based on the index update timeliness judgment result; after the index update timeliness judgment is qualified, performing a data warehouse multi-hop relationship query based on the qualified multi-level graph index, and determining whether to trigger a candidate node screening range shrinkage mechanism based on a multi-hop relationship result query accuracy verification result; after the multi-hop relationship result query accuracy verification is qualified, determining whether to trigger a query response optimization mechanism based on a data query response timeliness judgment result.

[0008] On the other hand, a data warehouse multi-hop relationship rapid retrieval system based on graph embedded index is provided. The system applies a data warehouse multi-hop relationship rapid retrieval method based on graph embedded index. The system includes: a data flow interference monitoring module, an index update timeliness monitoring module, a multi-hop relationship query accuracy monitoring module and a data query response timeliness monitoring module: wherein the data flow interference monitoring module is used to determine whether to trigger the interference suppression mechanism based on the data flow interference judgment result. After the data flow interference judgment is qualified, a graph model is modeled based on the obtained qualified preprocessed data to obtain a graph model. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization. The preprocessed data represents the collected multi-source data; the index update is timely The timeliness monitoring module is used to build a multi-level graph index based on qualified pre-processed data and the graph model after the graph model is built, and to judge the timeliness of the index update based on the real-time changes of the dynamic data flow, and to decide whether to trigger the index step update mechanism based on the result of the index update timeliness judgment; the multi-hop relationship query accuracy monitoring module is used to perform multi-hop relationship query in the data warehouse based on the qualified multi-level graph index after the index update timeliness judgment is qualified, and to decide whether to trigger the candidate node screening range contraction mechanism based on the result of the multi-hop relationship result query accuracy verification; the data query response timeliness monitoring module is used to decide whether to trigger the query response optimization mechanism based on the result of the data query response timeliness judgment after the multi-hop relationship result query accuracy verification is qualified.

[0009] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: 1. Determine whether to trigger the interference suppression mechanism based on the result of data flow interference judgment. The interference suppression mechanism is used to improve the accuracy of overall data processing and analysis. After the data flow interference judgment is qualified, the graph model is built based on the obtained qualified preprocessed data to obtain the graph model, which helps to reduce the interference of noise data and sudden peak flow on data cleaning standardization and entity association relationship, and lay a reliable data foundation for subsequent processing. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization. The preprocessed data represents the collected multi-source data. After the graph model is built, a multi-level graph index is constructed based on the qualified preprocessed data and the graph model. The multi-level graph index is used to improve the fast retrieval and accurate matching capabilities of multi-hop relationship queries in the data warehouse, and the index update timeliness is judged based on the real-time changes of the dynamic data flow. Based on the index update timeliness judgment result, it is decided whether to trigger the index step update mechanism, which helps to improve the timeliness and accuracy of graph index construction, and ensure that it can adapt to dynamic data flow changes. The step update mechanism is used to improve the synchronization of real-time changes between multi-level graph indexes and dynamic data streams. After the index update timeliness is judged to be qualified, a multi-hop relationship query of the data warehouse is performed based on the qualified multi-level graph index, and the decision on whether to trigger the candidate node screening range contraction mechanism is made based on the multi-hop relationship result query accuracy verification result. This helps to enhance the ability of complex multi-hop relationship queries to cope with dynamic data streams and nonlinear retrieval requirements, improve path traversal efficiency, and trigger the candidate node screening range contraction mechanism to improve the matching degree between multi-hop relationship results and query targets; after the multi-hop relationship result query accuracy verification is qualified, the decision on whether to trigger the query response optimization mechanism is made based on the data query response timeliness judgment result, which helps to reduce data query response delay and improve the data warehouse multi-hop relationship retrieval efficiency. The query response optimization mechanism is triggered to improve the real-time performance of data query processing, thereby improving the data warehouse multi-hop relationship retrieval efficiency and effectively solving the problem of low data warehouse multi-hop relationship retrieval efficiency caused by data flow interference in the existing technology.

[0010] 2. By performing harmonic averaging on the data flow interference data items to obtain the data flow interference index, it is helpful to comprehensively consider the joint impact of three key interference factors, namely noise data, burst peak flow and peak data within the window, avoid the deviation caused by single parameter evaluation, and make the data flow interference index more objectively reflect the overall degree to which the pre-processed data is affected by noise and peak flow. Judging whether to trigger the interference suppression mechanism based on the obtained data flow interference index helps to solve the problems of data cleaning standardization deviation and entity association relationship misleading caused by noise data, burst peak flow and peak data in a targeted manner. Through the interference suppression mechanism, data flow interference is suppressed in time, and unqualified pre-processed data is prevented from entering the subsequent graph model modeling process, ensuring the accuracy and reliability of the pre-processed data in the subsequent processing process, thereby providing a high-quality data foundation for multi-hop relationship retrieval in the data warehouse.

[0011] 3. In the high-load scenario of user behavior data processing during the promotion period of the e-commerce platform, the scale of data flow generated by the concurrent operations of massive users grows exponentially, and the data types are complex and diverse. At the same time, it is mixed with duplicate data caused by network fluctuations, false behavior data generated by malicious order brushing, and data transmission disorder caused by instantaneous traffic peaks. These factors are superimposed on each other and can easily cause data flow interference. It is necessary to judge the data flow interference. The result of the data flow interference judgment determines whether to trigger the interference suppression mechanism. The interference suppression mechanism means sending prompts to preset personnel to re-collect and preprocess data. Data collection and preprocessing means collecting multi-source data from multi-source systems, and performing data cleaning, data standardization and data noise reduction on the collected multi-source data in sequence, which helps to effectively identify data flow interference and solve various types of interference in the data flow in a targeted manner. Specifically, in high-load scenarios such as e-commerce platform promotions, it can effectively deal with the chain effects of noise data, sudden peak flow data and three-level false correlation data, and avoid processing delays caused by redundant noise occupying storage resources, information confusion caused by data transmission congestion aggravating correlation errors and other problems. The interference suppression mechanism is triggered by the data flow interference judgment result, so that the preset personnel can process different levels of interference data according to priority, give priority to duplicate logs to release storage resources, divert sudden data flows to alleviate transmission pressure, verify and correct false associations to ensure data accuracy, thereby blocking the spread of interference, and providing a reliable data source for subsequent graph model modeling, multi-level graph index construction and other links, which helps to reduce the problem of low accuracy of pre-processed data caused by data flow interference in the system under high load conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 This is a flow chart of a method for fast retrieval of multi-hop relationships in a data warehouse based on graph embedded indexing provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the basic architecture of a method for rapid retrieval of multi-hop relationships in a data warehouse based on graph embedded indexing provided by an embodiment of the present invention; Figure 3 This is a logic diagram of an interference suppression mechanism of a data warehouse multi-hop relationship fast retrieval method based on graph embedded indexing provided by an embodiment of the present invention; Figure 4It is a structural diagram of a data warehouse multi-hop relationship rapid retrieval system based on graph embedded indexing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0015] like Figure 1 FIG. 1 is a flowchart of a method for fast retrieval of multi-hop relationships in a data warehouse based on graph embedded indexes according to an embodiment of the present invention. Figure 1 It can be seen that: As an embodiment of the first aspect, first, data flow interference monitoring: during the data warehouse data search process, whether to trigger the interference suppression mechanism is determined based on the data flow interference judgment result. The interference suppression mechanism is used to improve the accuracy of overall data processing and analysis. After the data flow interference is judged to be qualified, a graph model is built based on the obtained qualified preprocessed data to obtain a graph model. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization, and the preprocessed data represents the collected multi-source data. Data flow interference monitoring helps to filter noise data and sudden peak flows in multi-source data streams, reduce the impact of data flow interference on data processing, and ensure that the preprocessed data based on which the subsequent graph model modeling is based, such as the association data between user identification and corresponding browsed product identification, cross-purchase record data between different product categories, etc., have high accuracy and reliability.

[0016] Secondly, index update timeliness monitoring: After the graph model is built, a multi-level graph index is constructed based on qualified preprocessed data and the graph model. The multi-level graph index is used to improve the fast retrieval and precise matching capabilities of multi-hop relationship queries in the data warehouse, and the index update timeliness is judged based on the real-time changes of the dynamic data stream. Based on the index update timeliness judgment result, it is decided whether to trigger the index step update mechanism. The index step update mechanism is used to improve the synchronization of the multi-level graph index and the real-time changes of the dynamic data stream; through index update timeliness monitoring, it helps to ensure that the multi-level graph index can be updated in time with the changes in the dynamic data stream, ensure the timeliness and accuracy of the index, and provide strong support for efficient multi-hop relationship queries.

[0017] Next, multi-hop relationship query accuracy monitoring: After the index update timeliness is judged to be qualified, a multi-hop relationship query is performed on the data warehouse based on the qualified multi-level graph index, and based on the multi-hop relationship result query accuracy verification result, it is decided whether to trigger the candidate node screening range contraction mechanism. Triggering the candidate node screening range contraction mechanism is used to improve the matching degree between the multi-hop relationship results and the query target; multi-hop relationship query accuracy monitoring helps to improve the fit between the multi-hop relationship query results and the query target, reduce invalid queries caused by result deviation, and improve the timeliness of complex multi-hop relationship queries.

[0018] Finally, data query response timeliness monitoring: After the multi-hop relationship result query accuracy verification is passed, the query response optimization mechanism is triggered based on the data query response timeliness judgment result to improve the real-time performance of data query processing; data query response timeliness monitoring helps to speed up the response speed of multi-hop relationship queries in the data warehouse, reduce query latency, ensure that accurate query results can be quickly returned when facing dynamic data streams, and improve overall query efficiency.

[0019] It should be noted that before the design of the data warehouse multi-hop relationship rapid retrieval method based on graph embedded index provided in this application, a database for storing various types of setting data is established, which includes but is not limited to preset interference thresholds, preset slight interference thresholds, preset severity interference thresholds, preset delay thresholds, preset query qualification thresholds, preset confidence thresholds, preset response thresholds, etc., among which various preset values ​​are directly set by technical personnel.

[0020] In this embodiment, through the joint action of data flow interference monitoring, index update timeliness monitoring, multi-hop relationship query accuracy monitoring and data query response timeliness monitoring, they influence each other to form a complete dynamic optimization closed loop, which not only ensures the accuracy of data processing and the timeliness of index update in the process of multi-hop relationship retrieval in the data warehouse, but also improves the accuracy of query results and the timeliness of response, and effectively solves the problem of low efficiency of multi-hop relationship retrieval in the data warehouse caused by data flow interference; among them, data flow interference monitoring provides qualified preprocessing data and graph models for subsequent index update timeliness monitoring. If the data flow interference is not effectively suppressed, it will lead to If the preprocessed data is inaccurate, the multi-level graph index constructed based on it will also have deviations, which will affect the timeliness and accuracy of the index update; the results of the index update timeliness monitoring will directly affect the multi-hop relationship query accuracy monitoring effect. If the multi-level graph index cannot be updated in time with the dynamic data stream, its timeliness and accuracy will be insufficient, which will lead to a decrease in the fit between the multi-hop relationship query results and the query target, and an increase in invalid queries; the results of the multi-hop relationship query accuracy monitoring will affect the data query response timeliness monitoring. If the multi-hop relationship query results have large deviations, repeated query adjustments are required, which will extend the query response time and affect the timeliness of the data query response.

[0021] like Figure 2 As shown in FIG, a basic architecture diagram of a method for fast retrieval of multi-hop relations in a data warehouse based on graph embedding indexes provided by an embodiment of the present invention is shown in FIG. Figure 2It can be seen that: during the data search process of the data warehouse, data flow interference judgment is performed, and the data flow interference index is obtained to determine whether the data flow interference index is less than the preset interference threshold. If so, the graph model is built. Otherwise, the interference suppression mechanism is triggered. After the interference suppression mechanism ends, if the data flow interference judgment is still unqualified, an interference suppression failure alarm is sent. Otherwise, the graph model is built. After the graph model is built, a multi-level graph index is constructed, and the index update timeliness is judged based on the real-time changes of the dynamic data flow, and the index update timeliness evaluation index is obtained to determine whether the index update timeliness evaluation index is less than the preset delay threshold. If so, a data warehouse multi-hop relationship query is performed. Otherwise, the index step update mechanism is triggered. After the index step update mechanism ends, if the index update timeliness is still unqualified, a step update failure alarm is sent. Otherwise, the data warehouse is carried out. The system performs multi-hop relationship query on the database and obtains the query accuracy index of the multi-hop relationship result, and determines whether the query accuracy index of the multi-hop relationship result is greater than the preset query qualification threshold. If so, the data query response timeliness is judged. Otherwise, the candidate node screening range contraction mechanism is triggered. After the candidate node screening range contraction mechanism ends, if the multi-hop relationship result query accuracy index is still unqualified, an optimization failure alarm is sent. Otherwise, the data query response timeliness is judged and the data query response timeliness result is obtained. It is determined whether the data query response timeliness result is less than the preset response threshold. If so, the data query response timeliness is determined to be qualified and the current query processing strategy is maintained. Otherwise, the query response optimization mechanism is triggered. After the query response optimization mechanism ends, if the data query response timeliness judgment is still unqualified, a query response optimization failure alarm is sent. Otherwise, the data query response is continuously monitored.

[0022] Furthermore, the specific process of determining whether to trigger the interference suppression mechanism based on the data flow interference judgment result is as follows: obtaining data flow interference data items to reflect the severity of various interference factors in the data flow and their impact on data quality; data flow interference data items include noise data verification value within the window, burst peak flow verification value and peak data verification value; the noise data verification value is represented by the ratio of noise data in the preset interference detection time period and the preset noise data ratio, and then weighted with the noise data influence weight coefficient, wherein the preset interference detection time period represents the time period corresponding to the data flow interference judgment, and the ratio quantification represents the ratio operation, and the preset interference detection time period is monitored by the data quality monitoring instrument. The amount of redundant information repeated data and the total amount of data in the measurement time period are quantified as the proportion of redundant information repeated data and the total amount of data as the noise data proportion, and the preset noise data proportion is represented by the average value of the noise data proportion in the historical time period; the burst peak flow verification value is represented by the burst peak flow duration and the preset burst peak flow duration in the preset interference detection time period. The duration of the burst peak flow is represented by the continuous time when the amount of data corresponding to the pre-processed data collected in the preset interference detection time period monitored by the flow monitoring instrument exceeds the preset data amount threshold, and the preset data amount threshold is set in advance by the preset personnel. The duration of the preset burst peak flow is represented by the average value of the duration of the burst peak flow in the historical time period; the peak data verification value is represented by the peak data proportion in the preset interference detection time period and the preset peak data proportion, and the result of weighting processing with the peak data influence weight coefficient, wherein the amount of data monitored by the data flow monitoring module corresponding to the pre-processed data collected in the preset interference detection time period exceeds the preset data amount threshold and the total data amount is quantified as the peak data proportion; the result of harmonic averaging processing of the data flow interference data item is used as the data flow interference index for reflecting the degree to which the pre-processed data is affected by noise and peak flow. When the duration of the burst peak flow is When the peak data rate increases, the data processing center is limited in its data processing capacity under a long period of high load, and is unable to effectively divert and process the large amount of data that is pouring in, resulting in an increase in the peak data ratio. Since a large amount of accumulated peak data exceeds the processing threshold of the data verification mechanism, some data will have problems such as format disorder and information loss during transmission and storage, which will lead to an increase in the data noise ratio. The increase in the duration of the burst peak flow, the peak data ratio and the data noise ratio will further cause the data flow interference index to increase; Data flow interference index qualification verification: Determine whether the data flow interference index is less than the preset interference threshold, where the preset interference threshold is represented by the average value of the data flow interference index in the historical time period;If so, the corresponding preprocessed data is marked as qualified, and a graphical model is constructed based on the qualified preprocessed data to obtain the graph model and store it in the data warehouse. Otherwise, the interference suppression mechanism is triggered. Preprocessed data refers to data collected from multiple source systems. The collected multi-source data is sequentially subjected to data cleaning, structure mapping and type conversion, and data standardization. Data cleaning automatically detects and cleans redundant values, noise, abnormal data, and malformed data in the input data. For example, meaningless blank fields are removed from log data and missing values ​​are filled. Structure mapping and type conversion parses semi-structured data, such as XML (eXtensible Markup Language) files, and converts them into a form-like structure, using standard Schema logic for semantic alignment. Data standardization standardizes units, time formats, and encoding methods. For example, timestamps are formatted in UTC (Coordinated Universal Time).

[0023] It should be noted that the noise data impact weight coefficient, the burst peak flow duration impact weight coefficient and the peak data impact weight coefficient are used to reflect the degree of influence of the data flow interference data item on the data flow interference index. In this embodiment, there is a group of mapping groups obtained from the database, which includes a mapping set, which is pre-set by professional and technical personnel. The mapping relationship adopts a one-to-one or many-to-one form, so as to accurately reflect the degree of influence of different interference factors on the data flow interference index in different data scenarios.

[0024] Specifically, by establishing a mapping relationship between the quantitative results of the proportion of data flow interference data items and the corresponding weight influence values, and using a numerical range of 0-1 to represent the weight influence value proportion, wherein the standard interference factor parameters include the preset noise data proportion, the preset burst peak flow duration and the preset peak data proportion, and the weight influence value includes the noise data influence weight coefficient, the burst peak flow duration influence weight coefficient and the peak data influence weight coefficient; when the system receives the quantitative results of the proportion of data flow interference data items, it can quickly retrieve the corresponding weight influence value from the pre-established mapping group, and then accurately quantify the degree of influence of the data flow interference data items on the data flow interference index, wherein the noise data influence weight coefficient mainly acts on the quantitative results related to the noise data proportion, the burst peak flow duration influence weight coefficient focuses on the quantitative results related to the burst peak flow duration, and the peak data influence weight coefficient affects the quantification of the peak data proportion. The three cooperate with each other to constitute a comprehensive influence system on the data flow interference index, which improves the accuracy and adaptability of data flow interference judgment.

[0025] In this embodiment, through data flow interference judgment, the noise data, sudden peak flow and peak data key interference factors in the window can be fully integrated. By quantitatively analyzing each interference data item and forming a data flow interference index, the degree of interference of the pre-processed data can be accurately reflected. Combined with the preset interference threshold for qualification verification, it can effectively determine whether the interference suppression mechanism needs to be triggered. This process not only realizes the scientific evaluation of data flow interference, but also can take targeted measures to reduce the impact of interference, ensuring that the pre-processed data based on the subsequent graph model modeling is of high quality, laying a reliable data foundation for the entire data warehouse multi-hop relationship retrieval process, and improving the accuracy and stability of data processing and analysis.

[0026] like Figure 3 As shown in FIG, the interference suppression mechanism logic diagram of the data warehouse multi-hop relationship fast retrieval method based on graph embedded index provided by the embodiment of the present invention is composed of Figure 3 It can be seen that: during the implementation of the interference suppression mechanism, a judgment is first made based on the data flow interference index: if the data flow interference index is not less than the preset interference threshold and is less than the preset slight interference threshold, the corresponding pre-processed data is marked as first-level noise data; if the data flow interference index is not less than the preset slight interference threshold and is less than the preset severity interference threshold, the corresponding pre-processed data is marked as second-level burst peak flow data; if the data flow interference index is not less than the preset severity interference threshold, the corresponding pre-processed data is marked as third-level false correlation relationship data, and layered filtering is started for the first-level noise data, dynamic buffering and diversion are implemented for the second-level burst peak flow data, and rule verification and correction are performed for the third-level false correlation relationship data. After the interference suppression mechanism ends, the interference suppression mechanism qualification verification is performed. If the re-acquired data flow interference index is still not less than the preset interference threshold, an interference suppression failure alarm is sent. Otherwise, graphical model modeling is performed.

[0027] Furthermore, the specific process of the interference suppression mechanism is as follows: SS1, judgment based on the data flow interference index: if the data flow interference index is not less than the preset interference threshold and less than the preset slight interference threshold, the corresponding pre-processed data is marked as the first-level noise data; if the data flow interference index is not less than the preset slight interference threshold and less than the preset severity interference threshold, the corresponding pre-processed data is marked as the second-level burst peak flow data; if the data flow interference index is not less than the preset severity interference threshold, the corresponding pre-processed data is marked as the third-level false correlation relationship data, wherein the preset slight interference threshold and the preset severity interference threshold All are set in advance by the preset personnel; SS2, start layered filtering for the first-level noise data: when the first-level noise data is monitored, the preset deduplication algorithm is used to identify and retain unique records, and duplicate reported pre-processed data is deleted. Among them, the use of layered filtering for the first-level noise data helps to efficiently remove redundant data. The preset deduplication algorithm is set in advance by the preset personnel, such as the hash deduplication algorithm; SS3, implement dynamic buffering and diversion for the second-level burst peak flow data: when the second-level burst peak flow data is monitored, enable the temporary data buffer to store excess data and import it into the processing flow in batches, among which the second-level burst peak flow data is dynamically Buffering and diversion help to ensure the stability of the data processing process; SS4, rule verification and correction for three-level false association relationship data: when three-level false association relationship data is detected, based on the predefined business association rule library, the entity interaction records are verified one by one, and false associations that do not conform to the rules are eliminated. That is, there is a predefined business association rule library in the database. The detected three-level false association relationship data is automatically matched with the corresponding rules in the rule library according to the associated entity type for a single entity interaction record. If the corresponding rule is successfully matched, the data is retained, otherwise the record is eliminated. Among them, the three-level false association relationship data is verified and corrected. Row rule verification and correction help improve the authenticity of entity associations; SS5, interference suppression mechanism qualification verification: After the interference suppression mechanism ends, the data flow interference index is re-acquired. If the data flow interference index is still not less than the preset interference threshold, an interference suppression failure alarm is sent. Otherwise, the corresponding preprocessed data is marked as qualified preprocessed data, and a graph model is built based on the qualified preprocessed data to obtain a graph model, and stored in the data warehouse. Among them, the qualified preprocessed data and core nodes, dynamic interaction nodes and semantic enhancement nodes in the historical data are randomly divided into training sets, and the training set is input into the heterogeneous graph construction model fused by GNN (Graph Neural Network) for training. The obtained training model is used as the graph model, and the re-acquired qualified preprocessed data is input into the graph model to output the core nodes, dynamic interaction nodes and semantic enhancement nodes.

[0028] In this embodiment, the interference suppression mechanism helps to reduce the negative impact of different levels of data flow interference on data processing. Among them, since different degrees of data flow interference have different impacts on data quality, for example, the first-level noise data mainly interferes with the redundancy of repeatedly reported data, and the interference level is relatively light. The second-level burst peak flow data will impact the processing flow, and the interference level is second. The third-level false association relationship data will mislead entity associations, and the interference level is the most serious. Therefore, the data flow interference data is divided into three levels of data flow interference data and targeted three-level optimization measures are taken. This can reduce the waste of resources or incomplete processing caused by single processing. Combined with the qualification verification of the interference suppression mechanism, it can minimize the impact of different levels of interference on pre-processed data, thereby providing high-quality qualified pre-processed data for graph model modeling.

[0029] Furthermore, a multi-level graph index is constructed based on qualified preprocessed data and graph models, and the timeliness of index updates is judged based on the real-time changes of dynamic data streams. The specific process is as follows: a multi-level graph index is constructed based on qualified preprocessed data and graph models, wherein the multi-level graph index means that the nodes are first partitioned by type through the entity classification method and a first-level type index is created, and then the nodes and edges in the partition are mapped to low-dimensional vectors through the preset graph embedding algorithm, and a second-level similarity index is constructed based on the low-dimensional vector and the edge weight information is processed through the weight grading method to obtain a filtering index, and the filtering indexes at all levels are associated in sequence, and the edge weight information represents information reflecting the closeness of the relationship between entity nodes, wherein the preset graph embedding algorithm is set in advance by the preset personnel, such as the Node2Vec graph embedding algorithm; after the multi-level graph index is constructed, it is associated with the graph model in the data warehouse, and the timeliness of index updates is judged based on the real-time changes of dynamic data streams.

[0030] Specifically, the specific process of judging the timeliness of index updates is as follows: obtaining the index update timeliness evaluation index; after the multi-level graph index is built, the time from the re-acquired qualified pre-processed data being written into the data warehouse to the completion of the update of the multi-level graph index is monitored by a time series monitoring instrument as the index update timeliness evaluation index; the index update timeliness evaluation index is used to reflect the response speed and synchronization efficiency of the multi-level graph index to the changes in new data in the data warehouse; index update timeliness evaluation index qualification verification: judging whether the index update timeliness evaluation index is less than the preset delay threshold, where the preset delay threshold is represented by the average value of the index update timeliness evaluation index in the historical time period; if so, the corresponding multi-level graph index is marked as a qualified multi-level graph index, and a multi-hop relationship query of the data warehouse is performed, otherwise, the index step update mechanism is triggered.

[0031] In this embodiment, by constructing a multi-level graph index, an efficient retrieval basis is provided for multi-hop relationship queries in data warehouses, which helps to improve the speed and accuracy of queries. By judging the timeliness of index updates, it helps to promptly discover whether index updates are lagging, ensuring that the multi-level graph index can keep up with real-time data stream changes, avoiding query result deviations caused by index update lags, and improving the timeliness of multi-level graph index updates. The combination of the two not only constructs a high-quality multi-level graph index, but also ensures the dynamic adaptability of the multi-level graph index, thereby providing strong support for the timeliness of multi-hop relationship retrieval in data warehouses.

[0032] Furthermore, the specific process of the index step update mechanism is as follows: the index update timeliness evaluation index, the iteration step of the current index construction and the index association relationship coverage are input into the iteration step mapping set for dynamically optimizing the index update rhythm in the database to obtain the iteration step adjustment coefficient. Among them, when constructing a multi-level graph index, the interval length of each iterative index update is monitored by the index iteration monitoring instrument as the iteration step of the current index construction, and the result of quantifying the proportion of the actual number of entity association relationships contained in the multi-level graph index and the total number of all entity association relationships in the data warehouse is used as the index association relationship coverage. There is a set of iteration step mapping sets in the database to reflect the correspondence between the combination of the index update timeliness evaluation index, the iteration step of the current index construction and the index association relationship coverage, and the corresponding iteration step adjustment coefficient. Relationship; The iteration step length mapping set is set in advance by the preset personnel and stored in the database. For example, the iteration step length mapping set is gradually constructed through statistical analysis of historical index update data and iterative verification of preset algorithms (such as gradient descent algorithm, etc.): first, the specific combination parameters of index update timeliness evaluation indicators, current index construction iteration step length and index association relationship coverage in a large number of actual index update scenarios are extracted, and each parameter is assigned a weighted influence value based on the importance of index update efficiency. The actual iteration step length adjustment coefficient in the corresponding scenario is simultaneously recorded. Then, through correlation analysis (such as Spearman rank correlation coefficient analysis, etc.), abnormal correlation data is eliminated, and the specific combination of statistically significant index update timeliness evaluation indicators, current index construction iteration step length and index association relationship coverage and the corresponding relationship with the iteration step length adjustment coefficient are retained;Iteration step adjustment coefficient qualification test: Determine whether the iteration step adjustment coefficient is within the preset reasonable step threshold range, where the preset reasonable step threshold range is set in advance by the preset personnel, and the preset reasonable step threshold range includes both end points to reduce the situation where the match cannot be made due to a fixed single value. If the iteration step adjustment coefficient is greater than the upper limit of the preset reasonable step threshold range, it may cause excessive index jumps, resulting in omissions in some data updates or imbalanced index structures, affecting data retrieval efficiency and accuracy. If the iteration step adjustment coefficient is less than the lower limit of the preset reasonable step threshold range, it may cause the index to be adjusted too frequently, consume system resources, and reduce the overall performance of index updates. If so, the iteration step adjustment coefficient is marked as a qualified iteration step adjustment value, otherwise, a step abnormality alarm is sent. Based on the corresponding qualified iteration step adjustment value The index build iteration step is gradually reduced by using the amplitude as the adjustment step. This helps ensure the effectiveness of index updates while reducing resource waste or index instability caused by large one-time adjustments, and gradually improves the responsiveness of index updates to dynamic data streams. The index update timeliness evaluation indicator is continuously monitored, and execution is stopped when it is detected to be less than the preset delay threshold. The index build iteration step is no less than the preset minimum step threshold, where the preset minimum step threshold is set in advance by the design personnel. The index step update mechanism is verified to be qualified: after the index step update mechanism ends, if the newly obtained index update timeliness evaluation indicator is still no less than the preset delay threshold, a step update failure alert is issued. Otherwise, the corresponding multi-level graph index is marked as qualified, and multi-hop relationship queries in the data warehouse are performed.

[0033] In this embodiment, the index step update mechanism is used to help dynamically optimize the update rhythm of the multi-level graph index, so that the index can better adjust to the changes in the dynamic data flow. Among them, the use of a step-by-step iterative step adjustment method can not only accurately adapt to the changes in the dynamic data flow and improve the timeliness and response speed of the index update, but also avoid problems such as resource waste and index instability caused by excessive or insufficient adjustment. At the same time, the effectiveness of the index step update mechanism is guaranteed through qualification verification, which ultimately improves the timeliness and reliability of the multi-level graph index update and provides efficient support for multi-hop relationship queries in data warehouses.

[0034] Furthermore, the specific process of multi-hop relationship query in data warehouse is as follows: input the query starting node into the secondary similarity index mapping set, and obtain the embedded vector in the secondary similarity index corresponding to the partition, wherein the query starting node represents the initial entity object specified when the user initiates the query, which is the starting entity of the multi-hop relationship query. There is a set of secondary similarity index mapping sets in the database, which is used to reflect the association relationship between the query starting node and the embedded vector in the secondary similarity index of the corresponding partition; the secondary similarity index mapping set is set in advance by the preset personnel and stored in the database, for example, it is gradually constructed through statistical analysis of historical multi-hop relationship query data and iterative verification of preset algorithms (such as K nearest neighbor algorithm, etc.) Secondary similarity index mapping sets: First, extract the embedding vectors of query start nodes and corresponding secondary similarity indexes from a large number of actual query scenarios. The actual associations between query start nodes and embedding vectors in these scenarios are simultaneously recorded. Then, through correlation analysis (such as cosine similarity analysis), outlier associations are removed, retaining statistically significant associations between query start nodes and embedding vectors in the secondary similarity index. Based on the obtained preset hop count and a preset retrieval algorithm, candidate node vectors with the highest similarity to the embedding vector of the start node are retrieved. The preset hop count is set in advance by the retrieval personnel, and the preset retrieval algorithm, such as KD (K-Dimensional Tree) retrieval, is also set in advance by the retrieval personnel. After completing the expansion of the preset hop count, the candidate paths are mapped back to the original entity and relationship data in the data warehouse. Finally, qualified multi-hop relationship results are returned and the accuracy of the multi-hop relationship results is verified. Original entities are entity units in the data warehouse that carry specific business information and have not been processed by the preset graph embedding algorithm. Relationship data represents the association information formed between original entities during business interactions, reflecting the connections between different original entities.

[0035] Specifically, the specific process of multi-hop relationship result query accuracy verification is as follows: obtain the multi-hop relationship result query accuracy index; the multi-hop relationship result query accuracy index is quantitatively expressed by the proportion of the number of multi-hop relationship paths successfully matched based on the preset matching algorithm in the preset verification sample set and the total number of query paths, wherein the preset verification sample set is set in advance by the preset personnel, and the preset matching algorithm is set in advance by the preset personnel, such as the cosine similarity matching algorithm, the edit distance matching algorithm, etc.; the multi-hop relationship result query accuracy index is used to reflect the degree of consistency between the multi-hop relationship query result and the actual business association and to judge whether the query result is valid; multi-hop relationship result query accuracy index qualification verification: judge whether the multi-hop relationship result query accuracy index is greater than the preset query qualification threshold, wherein the preset query qualification threshold is represented by the average value of the multi-hop relationship result query accuracy index in the historical time period; if so, the multi-hop relationship result is marked as a qualified multi-hop relationship result, and stored in the query result cache, and the data query response timeliness is judged, otherwise, the candidate node screening range contraction mechanism is triggered.

[0036] In this embodiment, through the data warehouse multi-hop relationship query and multi-hop relationship result query accuracy verification, it is helpful to efficiently obtain qualified multi-hop relationship results, ensure the consistency of the query results with the actual business association, and improve the accuracy of the query results. The data warehouse multi-hop relationship query uses the secondary similarity index and the preset retrieval algorithm to quickly locate the candidate nodes, complete the multi-hop expansion and map back to the original data, thereby ensuring the query efficiency. The multi-hop relationship result query accuracy verification can timely identify inaccurate query results and trigger the optimization mechanism through quantitative indicators and multi-hop relationship result query accuracy indicator qualification verification, further improving the accuracy of the query results and laying a good foundation for the subsequent data query response timeliness judgment.

[0037] Furthermore, the specific process of the candidate node screening range contraction mechanism is as follows: the multi-hop relationship result query accuracy index, the candidate node number threshold retrieved by the current preset retrieval algorithm and the path correlation are input into the candidate node range mapping set for improving the candidate node screening accuracy, and the corresponding screening range contraction ratio is obtained, wherein the candidate node number threshold retrieved by the current preset retrieval algorithm is set in advance by the preset personnel, and the path correlation obtained by accumulating and calculating the edge weight information represents the closeness of the relationship between the entity nodes on the candidate path in the multi-hop relationship query. There is a set of candidate node range mapping sets in the database, which is used to reflect the multi-hop relationship result query accuracy index, the current preset retrieval algorithm retrieval, and the path correlation of the candidate node range. The correspondence between the combination of the candidate node quantity threshold and path relevance, and the corresponding screening range contraction ratio; the candidate node range mapping set is set in advance by the preset personnel and stored in the database. For example, the candidate node range mapping set is gradually constructed through statistical analysis of historical multi-hop relationship query data and iterative verification of preset algorithms (such as random forest regression, etc.): first extract a large number of multi-hop relationship result query accuracy indicators in actual query scenarios, the specific combination parameters of the candidate node quantity threshold and path relevance of the preset retrieval algorithm, and assign a weighted influence value based on the importance of the business scenario to each parameter, and simultaneously record the actual contraction ratio of the screening range in the corresponding scenario, and then through correlation analysis (such as Pearson correlation (Data analysis, etc.) to eliminate abnormal correlation data and retain the multi-hop relationship results with statistical significance. Query accuracy index, preset candidate node number threshold of retrieval algorithm and specific combination of path correlation and corresponding relationship with screening range contraction ratio; Screening range contraction ratio qualification test: judge whether the screening range contraction ratio is greater than the preset maximum screening threshold, where the preset maximum screening threshold is set in advance by the preset personnel; if so, send a screening range contraction ratio failure alarm, otherwise, mark the screening range contraction ratio as a qualified screening range contraction value; based on the amplitude corresponding to the qualified screening range contraction value as the adjustment step, gradually reduce the number of candidate nodes retrieved by the preset retrieval algorithm, which helps to ensure that On the premise of omitting key candidate nodes, nodes with low correlation with the query target are gradually eliminated to improve the matching accuracy between candidate nodes and query targets. Gradual adjustment can accurately identify and retain nodes closely related to the query target, while gradually stripping off redundant nodes to avoid misjudgments that may be caused by one-time screening, reduce invalid calculations and redundant results, and thus improve the accuracy of multi-hop relationship query results; continuously monitor the query accuracy index of the re-acquired multi-hop relationship results, and stop execution when it is monitored that the query accuracy index of the multi-hop relationship results is greater than the preset query qualification threshold; the number of executions of the candidate node screening range contraction mechanism does not exceed the preset maximum number of executions, where the preset maximum number of executions is set in advance by the preset personnel;Verification of the eligibility of the candidate node screening range reduction mechanism: After the candidate node screening range reduction mechanism is completed, if the multi-hop relationship result query accuracy index is still not greater than the preset query qualification threshold, an optimization failure alarm is issued. Otherwise, the multi-hop relationship result is marked as qualified, synchronously stored in the query result cache, and the data query response timeliness is judged.

[0038] In this embodiment, the candidate node screening range contraction mechanism helps to improve the accuracy of multi-hop relationship query results, reduce invalid queries and redundant data, and through the step-by-step candidate node screening range adjustment, it can not only accurately retain the nodes closely related to the query target, but also effectively eliminate irrelevant nodes, thereby improving the accuracy of multi-hop relationship query results. At the same time, through continuous monitoring and execution time limit guarantee mechanism to run efficiently, and finally through the qualification verification of the candidate node screening range contraction mechanism to ensure the rationality of the candidate node screening range contraction adjustment, providing reliable result support for the subsequent data query response timeliness judgment.

[0039] Furthermore, the specific process of determining whether to trigger the query response optimization mechanism based on the result of the data query response timeliness judgment is as follows: obtaining the original data items of the data query response; the original data items of the data query response include the data query average response time verification value, the data query longest response time verification value and the data query timeout ratio verification value; the data query average response time verification value is expressed by quantifying the proportion of the data query average response time and the preset average response time within the preset query time period, and then weighting it with the data query average response time influence weight coefficient, wherein the preset query time period represents the time period for judging the timeliness of the data query response, and the data query response monitoring instrument is used to monitor all data warehouse multi-hop relationship queries from the client The time from initiation to return of query results is taken as the average response time of data query, and the preset average response time is represented by the average value of the average response time of data query in the historical time period; the data query maximum response time verification value is represented by the result after quantifying the proportion of the data query maximum response time in the preset query time period and the preset maximum response time, and weighting them with the weight coefficient of the data query maximum response time, among which, the maximum time consumed for querying a single query data monitored by a single query time consumption monitoring instrument is taken as the longest response time of data query, and the preset maximum response time is represented by the average value of the longest response time of data query in the historical time period; the data query timeout ratio verification value is represented by the ratio of the data query timeout ratio in the preset query time period to the preset data query timeout ratio. After the query timeout ratio is quantified, the result after weighting processing with the data query timeout ratio influence weight coefficient is expressed, wherein the result of quantifying the ratio of the number of query data whose query data response time exceeds the preset response threshold to the total number of query data is used as the data query timeout ratio, and the preset data query timeout ratio is expressed by the average value of the data query timeout ratio in the historical time period; the result after harmonic averaging processing of the original data items of the data query response is used as the data query response timeliness result. When the average response time of the data query increases, due to the increase in the overall processing load of the data processing center and the intensification of resource competition (such as increased CPU usage, memory allocation delay, etc.), the processing resources of a single complex query or a high-priority query are divided. The spread leads to an increase in the longest response time of data queries. Since the probability of the longest response time of data queries exceeding the preset response threshold increases, and long-consuming queries will occupy system resources, causing other query processing delays, which in turn leads to an increase in the proportion of data query timeouts. The increase in the average response time of data queries, the longest response time of data queries, and the increase in the proportion of data query timeouts will further increase the timeliness of data query response results. The timeliness of data query response results is used to reflect the overall response speed and stability of multi-hop relationship data queries. The eligibility verification of data query response timeliness results: determine whether the data query response timeliness result is less than the preset response threshold, where the preset response threshold is represented by the average value of the data query response timeliness results in the historical time period.If so, the data query response timeliness is determined to be up to standard and the current query processing strategy is maintained. Otherwise, the query response optimization mechanism is triggered. The query processing strategy indicates that during the multi-hop relationship query process of the data warehouse, when the data query response timeliness is up to standard, the original multi-level graph index-based query method is maintained to execute the data query.

[0040] Specifically, the specific process of the query response optimization mechanism is as follows: retrieve all associated paths and their corresponding path confidences contained in the qualified multi-hop relationship results from the data warehouse, and determine whether the path confidence is less than a preset confidence threshold, where the preset confidence threshold is represented by the average value of the path confidence in the historical time period; if so, mark the corresponding associated path as a redundant path and directly delete the redundant path; otherwise, retain the corresponding associated path as a valid associated path; query response optimization mechanism qualification verification: after the query response optimization mechanism ends, re-obtain the data query response timeliness result. If the data query response timeliness result is still not less than the preset response threshold, send a query response optimization failure alarm; otherwise, perform continuous monitoring of the data query response.

[0041] It should be noted that the weight coefficient of the average response time of data query, the weight coefficient of the longest response time of data query and the weight coefficient of the data query timeout ratio are used to reflect the degree of influence of the original data items of the data query response on the timeliness result of the data query response. In this embodiment, there is a group of mapping groups obtained from the database, which includes a mapping set, which is pre-set by professional and technical personnel. The mapping relationship adopts a one-to-one or many-to-one form, so as to accurately reflect the degree of influence of different data query response factors on the timeliness result of the data query response in different query scenarios. Specifically, by establishing a mapping relationship between the quantitative results of the proportion of the original data items in the data query response and the corresponding weight influence value, and using the 0-1 numerical range to represent the weight influence value proportion, among which the standard query response parameters include the preset average response time, the preset maximum response time and the preset data query timeout ratio, and the weight influence value includes the weight coefficient of the average response time of the data query, the weight coefficient of the maximum response time of the data query and the weight coefficient of the data query timeout ratio; when the system receives the quantitative results of the proportion of the original data items in the data query response, it can quickly retrieve the corresponding weight from the pre-built mapping group The impact value is used to accurately quantify the degree of influence of the original data items of the data query response on the timeliness of the data query response. Among them, the weight coefficient of the average response time of the data query mainly affects the quantitative results related to the average response time of the data query. The weight coefficient of the longest response time of the data query focuses on the quantitative results related to the longest response time of the data query. The weight coefficient of the data query timeout ratio affects the quantification of the data query timeout ratio. The three work together to form a comprehensive influence system on the timeliness of the data query response, which improves the accuracy and adaptability of the judgment of the timeliness of the data query response.

[0042] In this embodiment, the query response timeliness judgment and query response optimization mechanism are used to help improve the efficiency of multi-hop relationship queries in data warehouses. The data query response timeliness judgment is based on multi-dimensional considerations of the original data items of the data query response, and can objectively reflect the overall performance of multi-hop relationship queries, providing reasonable quantitative indicators for triggering the query response optimization mechanism. The query response optimization mechanism effectively reduces invalid data processing by eliminating redundant association paths with low confidence, and significantly improves the query response speed and stability. Through the effective combination of query response timeliness judgment and query response optimization mechanism, it not only ensures the stability of the query processing strategy, but also can timely intervene in the query response optimization mechanism when the query response timeliness judgment is unqualified, thereby achieving an improvement in the efficiency of multi-hop relationship queries in data warehouses.

[0043] like Figure 4As shown, it is a structural diagram of the data warehouse multi-hop relationship rapid retrieval system based on graph embedded index provided by an embodiment of the present application. The data warehouse multi-hop relationship rapid retrieval system based on graph embedded index provided by an embodiment of the present application applies a data warehouse multi-hop relationship rapid retrieval method based on graph embedded index, which is characterized by including: a data flow interference monitoring module, an index update timeliness monitoring module, a multi-hop relationship query accuracy monitoring module and a data query response timeliness monitoring module: wherein, the data flow interference monitoring module is used to determine whether to trigger the interference suppression mechanism based on the data flow interference judgment result, after the data flow interference judgment is qualified, a graph model is modeled based on the obtained qualified preprocessed data to obtain a graph model, and the qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization. , the preprocessed data represents the collected multi-source data; the index update timeliness monitoring module is used to build a multi-level graph index based on qualified preprocessed data and graph model after the graph model is built, and to judge the index update timeliness based on the real-time changes of the dynamic data flow, and to decide whether to trigger the index step update mechanism based on the index update timeliness judgment result; the multi-hop relationship query accuracy monitoring module is used to perform data warehouse multi-hop relationship query based on qualified multi-level graph index after the index update timeliness judgment is qualified, and to decide whether to trigger the candidate node screening range contraction mechanism based on the multi-hop relationship result query accuracy verification result; the data query response timeliness monitoring module is used to decide whether to trigger the query response optimization mechanism based on the data query response timeliness judgment result after the multi-hop relationship result query accuracy verification is qualified.

[0044] In this embodiment, the data flow interference monitoring module, the index update timeliness monitoring module, the multi-hop relationship query accuracy monitoring module and the data query response timeliness monitoring module present a close collaborative relationship of progressive layers and mutual support. The data flow interference monitoring module obtains qualified preprocessed data through data cleaning and standardization and data flow interference suppression, and provides high-quality qualified preprocessed data for subsequent graph model modeling. The index update timeliness monitoring module, based on the data flow interference monitoring module, constructs and dynamically optimizes the multi-level graph index to ensure that the multi-level graph index can keep up with data changes and provide an efficient retrieval basis for multi-hop relationship queries. Its judgment result on the timeliness of index updates directly affects whether the multi-hop relationship query accuracy monitoring module can be based on the qualified multi-level graph index. The query operation is carried out, and the multi-hop relationship query accuracy monitoring module relies on the qualified multi-level graph index to execute the query, and the accuracy of the query result is guaranteed by the multi-hop relationship result query accuracy verification. When the multi-hop relationship result query accuracy verification fails, the candidate node screening range contraction mechanism is triggered, and the qualified multi-hop relationship result output by the data query response timeliness monitoring module is the basis for response judgment. The data query response timeliness monitoring module further optimizes the query response speed on the basis of the qualified multi-hop relationship result query accuracy verification, ensuring that the system can return results quickly and stably. The four modules are linked together to form a complete closed-loop system, which jointly guarantee the efficiency, accuracy and timeliness of rapid retrieval of multi-hop relationships in data warehouses based on graph embedded indexes.

[0045] As an embodiment of the second aspect, during the data warehouse data search process, data flow interference judgment is performed, and the data flow interference index is obtained to determine whether the data flow interference index is less than the preset interference threshold. If so, graph model building is performed. Otherwise, the interference suppression mechanism is triggered. The interference suppression mechanism indicates that a prompt is sent to the preset personnel to re-collect and preprocess the data. After the interference suppression mechanism ends, if the data flow interference judgment is still unqualified, an interference suppression failure alarm is sent. Otherwise, graph model building is performed. After the graph model building is completed, a multi-level graph index is constructed, and the index update timeliness is judged based on the real-time changes of the dynamic data flow, and the index update timeliness evaluation index is obtained to determine whether the index update timeliness evaluation index is less than the preset delay threshold. If so, a data warehouse multi-hop relationship query is performed. Otherwise, the index step update mechanism is triggered. After the index step update mechanism ends, if the index update timeliness is still unqualified, an alarm is issued. Send a step update failure alarm. Otherwise, perform a multi-hop relationship query on the data warehouse, and obtain the multi-hop relationship result query accuracy index to determine whether the multi-hop relationship result query accuracy index is greater than the preset query qualification threshold. If so, perform a data query response timeliness judgment. Otherwise, trigger the candidate node screening range contraction mechanism. After the candidate node screening range contraction mechanism ends, if the multi-hop relationship result query accuracy index is still unqualified, send an optimization failure alarm. Otherwise, perform a data query response timeliness judgment, and obtain the data query response timeliness result to determine whether the data query response timeliness result is less than the preset response threshold. If so, determine that the data query response timeliness meets the standard and maintain the current query processing strategy. Otherwise, trigger the query response optimization mechanism. After the query response optimization mechanism ends, if the data query response timeliness judgment is still unqualified, send a query response optimization failure alarm. Otherwise, perform continuous monitoring of the data query response.

[0046] Based on the first aspect of the embodiment, in the high-load scenario of user behavior data processing during the e-commerce platform promotion period, the platform not only has to cope with the explosive growth of user browsing, ordering and other behavioral data in a short period of time, but also needs to synchronously process product information updates, inventory change records and other data. The server computing power and data transmission channels are both in a high-load state: a large number of repeated click logs occupy storage resources, resulting in processing delays in sudden order data streams; and the information confusion caused by data transmission congestion will aggravate the generation of false association relationship data, forming a chain problem from noise redundancy to processing delays, and then to association errors.

[0047] In this embodiment, in high-load scenarios for user behavior data processing during a major promotion on an e-commerce platform, such as a surge in user visits, order requests, and product search frequency, data flow interference is determined and a data flow interference index is obtained. If the data flow interference index exceeds a preset interference threshold, triggering an interference suppression mechanism, the e-commerce data management platform will execute intervention measures in order of priority. First, for noise data, preset personnel are prompted to prioritize processing repeated user click logs, and redundant data is stored separately by temporarily expanding the storage partition to avoid occupying the core data processing channel. Next, for sudden peak flow data, the order data flow of popular products is diverted, and processing queues are allocated according to user levels to reduce the impact of high concurrency on the data transmission channel. Finally, for false association relationship data, personnel are guided to perform batch verification of incorrectly associated delivery addresses and payment accounts, and a predefined business association rule library is used to quickly correct the mapping relationship to ensure that the accuracy of key transaction data is prioritized. This helps to promptly block the chain reaction of data flow interference and ensure the stability and reliability of data processing on the e-commerce platform during the major promotion.

[0048] In summary, the embodiment of the present application determines whether to trigger the interference suppression mechanism based on the result of data flow interference judgment. The interference suppression mechanism is used to improve the accuracy of overall data processing and analysis. After the data flow interference judgment is qualified, a graph model is built based on the obtained qualified preprocessed data to obtain a graph model, which helps to reduce the interference of noise data and sudden peak flows on data cleaning standardization and entity association relationships, and lays a reliable data foundation for subsequent processing. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization. The preprocessed data represents the collected multi-source data. After the graph model modeling is completed, a multi-level graph index is constructed based on the qualified preprocessed data and the graph model. The multi-level graph index is used to improve the ability of fast retrieval and precise matching of multi-hop relationship queries in data warehouses, and to judge the timeliness of index updates based on real-time changes in dynamic data streams. Based on the index update timeliness judgment result, it is decided whether to trigger the index step update mechanism, which helps to improve the timeliness and accuracy of graph index construction and ensure that it can adapt to dynamic data stream changes. The index step update mechanism is used to improve the real-time change synchronization between the multi-level graph index and the dynamic data flow. After the index update timeliness is judged to be qualified, the data warehouse multi-hop relationship query is performed based on the qualified multi-level graph index, and the candidate node screening range contraction mechanism is triggered based on the multi-hop relationship result query accuracy verification result. This helps to enhance the ability of complex multi-hop relationship queries to cope with dynamic data flows and nonlinear retrieval requirements, improve path traversal efficiency, and trigger the candidate node screening range contraction mechanism to improve the matching degree between the multi-hop relationship results and the query target; after the multi-hop relationship result query accuracy verification is qualified, the query response optimization mechanism is triggered based on the data query response timeliness judgment result, which helps to reduce the data query response delay and improve the data warehouse multi-hop relationship retrieval efficiency. The query response optimization mechanism is triggered to improve the real-time performance of data query processing, thereby realizing the improvement of the data warehouse multi-hop relationship retrieval efficiency and effectively solving the problem of low efficiency of data warehouse multi-hop relationship retrieval caused by data flow interference in the existing technology.

[0049] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0050] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

Claims

1. A fast retrieval method for multi-hop relations in a data warehouse based on graph embedding index, characterized by: include: During the data warehouse data search process, a decision is made on whether to trigger an interference suppression mechanism based on the data flow interference judgment result. The interference suppression mechanism is used to improve the accuracy of overall data processing and analysis. After the data flow interference judgment is qualified, a graph model is built based on the obtained qualified pre-processed data to obtain a graph model. The qualified pre-processed data represents the pre-processed data obtained after cleaning and standardization, and the pre-processed data represents the collected multi-source data. After the graph model is built, a multi-level graph index is constructed based on qualified pre-processed data and the graph model, and the timeliness of index update is judged based on the real-time changes of the dynamic data stream. Based on the result of the index update timeliness judgment, it is decided whether to trigger the index step update mechanism. The multi-level graph index is used to improve the fast retrieval and accurate matching capabilities of multi-hop relationship queries in the data warehouse, and the index step update mechanism is used to improve the synchronization of the multi-level graph index with the real-time changes of the dynamic data stream; After the index update timeliness is judged to be qualified, a multi-hop relationship query of the data warehouse is performed based on the qualified multi-level graph index, and a decision is made based on the accuracy check result of the multi-hop relationship result query whether to trigger the candidate node screening range contraction mechanism. The triggering of the candidate node screening range contraction mechanism is used to improve the matching degree between the multi-hop relationship result and the query target; After the multi-hop relationship result query accuracy verification is qualified, it is determined whether to trigger the query response optimization mechanism based on the data query response timeliness judgment result. The triggered query response optimization mechanism is used to improve the real-time performance of data query processing.

2. The method for fast retrieval of multi-hop relationships in a data warehouse based on graph embedded index according to claim 1 is characterized in that: The specific process of determining whether to trigger the interference suppression mechanism based on the data flow interference judgment result is as follows: Obtaining data flow interference data items to reflect the severity of various interference factors in the data flow and their impact on data quality. The data flow interference data items include a noise data verification value within a window, a burst peak flow verification value, and a peak data verification value; The noise data verification value is represented by the result of weighting the noise data proportion within the preset interference detection time period and the preset noise data proportion, and the noise data influence weight coefficient. The noise data proportion is represented by the result of weighting the redundant information repeated data amount and the total data amount within the preset interference detection time period. The burst peak flow verification value is represented by a result of weighting the burst peak flow duration and the preset burst peak flow duration within the preset interference detection time period, and the weight coefficient of the burst peak flow duration is used to determine the proportion of the burst peak flow duration and the preset burst peak flow duration. The burst peak flow duration is represented by the continuous duration during which the amount of data corresponding to the preprocessed data collected within the preset interference detection time period exceeds a preset data amount threshold. The peak data verification value is represented by the result of weighting the peak data proportion within the preset interference detection time period and the preset peak data proportion, and the result of weighting the peak data influence weight coefficient. The peak data proportion is represented by the result of quantifying the proportion of the data volume corresponding to the preprocessed data collected within the preset interference detection time period that exceeds the preset data volume threshold and the total data volume; The result of harmonic mean processing of the data flow interference data item is used as the data flow interference index for reflecting the degree to which the pre-processed data is affected by noise and peak flow; Determine whether the data flow interference index is less than the preset interference threshold. If so, mark the corresponding preprocessed data as qualified preprocessed data, and perform graph modeling based on the qualified preprocessed data to obtain the graph model and store it in the data warehouse. Otherwise, trigger the interference suppression mechanism.

3. The method for fast retrieval of multi-hop relationships in a data warehouse based on graph embedded index according to claim 2 is characterized in that: The interference suppression mechanism means sending a prompt to a preset person to re-collect and pre-process the data.

4. The method for fast retrieval of multi-hop relationships in a data warehouse based on graph embedded index according to claim 2 is characterized in that: The specific process of the interference suppression mechanism is as follows: SS1, based on the data flow interference index, judgment is made: if the data flow interference index is not less than the preset interference threshold and less than the preset mild interference threshold, the corresponding pre-processed data is marked as level 1 noise data; if the data flow interference index is not less than the preset mild interference threshold and less than the preset severe interference threshold, the corresponding pre-processed data is marked as level 2 burst peak flow data; if the data flow interference index is not less than the preset severe interference threshold, the corresponding pre-processed data is marked as level 3 false correlation relationship data; SS2, when level 1 noise data is detected, unique records are identified and retained using a preset deduplication algorithm, and duplicate reporting of pre-processed data is deleted; SS3, when detecting secondary burst peak flow data, a temporary data buffer is enabled to store excess data and import it into the processing flow in batches; SS4, when detecting third-level false association data, verifies entity interaction records one by one based on the predefined business association rule library and eliminates false associations that do not comply with the rules; SS5, verification of the qualification of the interference suppression mechanism: After the interference suppression mechanism ends, the data flow interference index is re-obtained. If the data flow interference index is still not less than the preset interference threshold, an interference suppression failure alarm is sent. Otherwise, the corresponding preprocessed data is marked as qualified preprocessed data, and a graph model is obtained based on the qualified preprocessed data and stored in the data warehouse.

5. The method for fast retrieval of multi-hop relations in a data warehouse based on graph embedded index according to claim 1 is characterized in that: The multi-level graph index is constructed based on qualified pre-processed data and graph models, and the timeliness of index updates is judged based on the real-time changes of dynamic data streams. The specific process is as follows: Construct a multi-level graph index based on qualified preprocessed data and a graph model. The multi-level graph index means first partitioning the nodes by type through an entity classification method and creating a first-level type index. Then, using a preset graph embedding algorithm, the nodes and edges in the partition are mapped into low-dimensional vectors. Based on the low-dimensional vectors, a second-level similarity index is constructed. The edge weight information is processed through a weight grading method to obtain a filtering index, and the filtering indexes at each level are sequentially associated. After the multi-level graph index is built, it is associated with the graph model in the data warehouse, and the timeliness of index updates is determined based on the real-time changes in dynamic data streams. Obtain an index update timeliness evaluation index, which is expressed as the time from when the re-acquired qualified pre-processed data is written into the data warehouse to when the multi-level graph index is updated after the multi-level graph index is built. The index update timeliness evaluation index is used to reflect the response speed and synchronization efficiency of the multi-level graph index to new data changes in the data warehouse; Determine whether the index update timeliness evaluation index is less than the preset delay threshold. If so, mark the corresponding multi-level graph index as a qualified multi-level graph index and perform a multi-hop relationship query on the data warehouse. Otherwise, trigger the index step update mechanism.

6. The method for fast retrieval of multi-hop relations in a data warehouse based on graph embedded index according to claim 5 is characterized in that: The specific process of the index step update mechanism is as follows: Input the index update timeliness evaluation index, the iteration step of the current index construction, and the index association relationship coverage into the iteration step mapping set for dynamically optimizing the index update rhythm in the database to obtain the iteration step adjustment coefficient; Determine whether the iteration step adjustment coefficient is within the preset reasonable step threshold range. If so, mark the iteration step adjustment coefficient as a qualified iteration step adjustment value. Otherwise, send a step abnormality alarm. Based on the amplitude corresponding to the qualified iteration step adjustment value as the adjustment step, the index construction iteration step is gradually reduced, and the index update timeliness evaluation index is continuously monitored. When the index update timeliness evaluation index is monitored to be less than the preset delay threshold, the execution is stopped; When the index step update mechanism ends, if the re-acquired index update timeliness evaluation index is still not less than the preset delay threshold, a step update failure alarm will be sent; otherwise, the corresponding multi-level graph index will be marked as a qualified multi-level graph index, and a data warehouse multi-hop relationship query will be performed.

7. The method for fast retrieval of multi-hop relations in a data warehouse based on graph embedded index according to claim 6 is characterized in that: The specific process of multi-hop relationship query in the data warehouse is as follows: Input the query starting node into the secondary similarity index mapping set, obtain the embedding vector in the secondary similarity index corresponding to the partition, and retrieve the candidate node vector with the highest similarity to the embedding vector of the starting node based on the obtained preset hop count and the preset retrieval algorithm; After completing the expansion of the preset number of hops, the candidate path is mapped back to the original entity and relationship data in the data warehouse, and finally the multi-hop relationship results that meet the conditions are returned for multi-hop relationship result query accuracy verification. The original entity represents the entity unit in the data warehouse that carries specific business information and has not been processed by the preset graph embedding algorithm. The relationship data represents the association information formed between the original entities in the business interaction, which is used to reflect the connection between different original entities. The multi-hop relationship result query accuracy verification is performed by obtaining a multi-hop relationship result query accuracy index, which is quantitatively expressed by the ratio of the number of multi-hop relationship paths successfully matched based on the preset matching algorithm and the total number of query paths in the preset verification sample set. The multi-hop relationship result query accuracy index is used to reflect the degree of consistency between the multi-hop relationship query result and the actual business association and to determine whether the query result is valid; Determine whether the multi-hop relationship result query accuracy index is greater than the preset query qualification threshold. If so, mark the multi-hop relationship result as a qualified multi-hop relationship result, store it in the query result cache, and perform data query response timeliness judgment. Otherwise, trigger the candidate node screening range contraction mechanism.

8. The method for fast retrieval of multi-hop relations in a data warehouse based on graph embedded index according to claim 7, characterized in that: The specific process of the candidate node screening range narrowing mechanism is as follows: The multi-hop relationship result query accuracy index, the number threshold of candidate nodes retrieved by the current preset retrieval algorithm, and the path correlation are input into a candidate node range mapping set for improving the candidate node screening accuracy, and the corresponding screening range contraction ratio is obtained; Determine whether the screening range shrinkage ratio is greater than a preset maximum screening threshold, and if so, send a screening range shrinkage ratio failure alarm; otherwise, mark the screening range shrinkage ratio as a qualified screening range shrinkage value; Based on the amplitude corresponding to the qualified screening range contraction value as the adjustment step, the number of candidate nodes retrieved by the preset retrieval algorithm is gradually reduced, and the query accuracy index of the re-obtained multi-hop relationship result is continuously monitored. When the multi-hop relationship result query accuracy index is monitored to be greater than the preset query qualification threshold, the execution is stopped; After the candidate node screening range shrinking mechanism ends, if the multi-hop relationship result query accuracy index is still not greater than the preset query qualification threshold, an optimization failure alarm is sent. Otherwise, the multi-hop relationship result is marked as a qualified multi-hop relationship result, synchronously stored in the query result cache, and the data query response timeliness is judged.

9. The method for fast retrieval of multi-hop relations in a data warehouse based on graph embedded index according to claim 1, characterized in that: The decision on whether to trigger the query response optimization mechanism is based on the result of the data query response timeliness judgment. The specific process is as follows: Obtaining raw data items of data query responses, including verification values ​​of the average response time of data queries, the longest response time of data queries, and the timeout ratio of data queries; The data query average response time verification value is expressed by quantifying the ratio of the data query average response time within the preset query time period to the preset average response time, and then weighting it with the data query average response time influence weight coefficient. The data query average response time is quantified by quantifying the ratio of the sum of the time from the client initiating all data warehouse multi-hop relationship queries to the time the query results are returned to the total amount of query target data; The data query maximum response time verification value is expressed by quantifying the ratio of the data query maximum response time within the preset query time period to the preset maximum response time, and then weighting it with the data query maximum response time influence weight coefficient. The data query maximum response time is expressed by the maximum time consumed for querying a single query data. The data query timeout ratio verification value is expressed by quantifying the ratio of the data query timeout ratio within the preset query time period to the preset data query timeout ratio, and then weighting them with the data query timeout ratio influence weight coefficient. The data query timeout ratio is quantified by the ratio of the number of query data whose query data response time exceeds the preset response threshold to the total number of query data; The result of harmonic mean processing of the original data items of the data query response is used as the data query response timeliness result for reflecting the overall response speed and stability of the multi-hop relationship data query; Determine whether the data query response timeliness result is less than the preset response threshold. If so, the data query response timeliness is determined to be up to standard and the current query processing strategy is maintained. Otherwise, the query response optimization mechanism is triggered. The specific process of the query response optimization mechanism is as follows: all associated paths and their corresponding path confidences contained in the qualified multi-hop relationship results are retrieved from the data warehouse, and the path confidence is determined to be less than a preset confidence threshold. If so, the corresponding associated path is marked as a redundant path and directly deleted. Otherwise, the corresponding associated path is retained as a valid associated path. After the query response optimization mechanism ends, the data query response timeliness result is obtained again. If the data query response timeliness result is still not less than the preset response threshold, a query response optimization failure alarm is sent. Otherwise, continuous monitoring of data query response is carried out.

10. A data warehouse multi-hop relationship fast retrieval system based on graph embedded index, applying a data warehouse multi-hop relationship fast retrieval method based on graph embedded index according to any one of claims 1 to 9, characterized in that: include: Data flow interference monitoring module, index update timeliness monitoring module, multi-hop relationship query accuracy monitoring module and data query response timeliness monitoring module: The data flow interference monitoring module is used to determine whether to trigger the interference suppression mechanism based on the data flow interference judgment result. After the data flow interference is judged to be qualified, a graph model is modeled based on the obtained qualified preprocessed data to obtain a graph model. The qualified preprocessed data represents the preprocessed data obtained after cleaning and standardization, and the preprocessed data represents the collected multi-source data. The index update timeliness monitoring module is used to build a multi-level graph index based on qualified preprocessed data and the graph model after the graph model is built, and to judge the timeliness of index updates based on the real-time changes of dynamic data streams, and to decide whether to trigger the index step update mechanism based on the result of the index update timeliness judgment; The multi-hop relationship query accuracy monitoring module is used to perform a multi-hop relationship query in the data warehouse based on the qualified multi-level graph index after the index update timeliness is judged to be qualified, and to determine whether to trigger the candidate node screening range contraction mechanism based on the multi-hop relationship result query accuracy verification result; The data query response timeliness monitoring module is used to determine whether to trigger a query response optimization mechanism based on a data query response timeliness judgment result after the multi-hop relationship result query accuracy verification is qualified.

Images