Method and system for centralized query based on data decoupling triggered by mq message

Abstract

The application provides a data decoupling centralized query method and system based on MQ message triggering, relates to the technical field of data query, and comprises the following steps: receiving a message queue data change message, constructing a field-dependent directed acyclic graph to divide a check node, verifying data consistency, pushing a transaction identifier to perform concurrent operation scheduling, collecting hotspot information by using a Bloom filter to adjust data sharding and cache distribution, generating a distributed query routing table, and distributing a query request according to the routing table. The application improves data consistency and optimizes query performance.

Description

TECHNICAL FIELD

[0001] The present application relates to data query technology, in particular to a data decoupling centralized query method and system based on MQ message triggering. BACKGROUND

[0002] With the deepening development of enterprise informatization, the business systems of large organizations are becoming increasingly complex, and the demand for data exchange and sharing between various business subsystems is growing. In distributed system architecture, message queues are widely used in data synchronization and interaction between systems, realizing the decoupling of producers and consumers, and improving the scalability and fault tolerance of the system.

[0003] However, the prior art has some obvious deficiencies in processing data query based on message queue triggering: the prior art lacks effective guarantee mechanism for data consistency, especially in high concurrency environment, it is difficult to ensure the correct order and integrity of data change operation, and it is easy to cause data inconsistency problem. The query performance optimization of the prior art is relatively static, and cannot dynamically adjust resource allocation according to the actual data access mode, resulting in performance bottleneck when the data access hot spot changes. The prior art lacks flexible query plan adjustment mechanism when processing complex queries, and when the query execution encounters performance problems, it is difficult to adaptively adjust the query strategy, resulting in unstable query response time.

[0004] With the continuous growth of the data volume and concurrent access volume of the business system, there is an urgent need for a data decoupling query method that can guarantee data consistency, has dynamic optimization capability, and can efficiently respond to query requests, to meet the needs of modern enterprises for high-performance and high-reliability data services. SUMMARY

[0005] The embodiment of the present application provides a data decoupling centralized query method and system based on MQ message triggering, which can solve the problems in the prior art.

[0006] The first aspect of the embodiment of the present application provides a data decoupling centralized query method based on MQ message triggering, comprising:

[0007] receiving data change messages from the message queue;

[0008] constructing field dependency relationship of a data table as a directed acyclic graph, dividing nodes into real-time check nodes and delayed check nodes based on data integrity requirements, performing data consistency verification on the real-time check nodes, performing asynchronous consistency verification on the delayed check nodes, triggering data correction and re-performing consistency verification when detecting that node data is inconsistent, pushing transaction identifiers carrying timestamps after consistency verification, the transaction identifiers using unique identification codes to mark the data change messages, performing priority identification on concurrent data change operations based on the transaction identifiers in combination with scheduling of a token bucket algorithm, and storing the data change operations that pass verification and the corresponding transaction identifiers to a central database;

[0009] collecting data access hotspot information of the central database using a Bloom filter, dynamically adjusting data sharding and cache distribution according to the data access hotspot information, and generating a distributed query routing table;

[0010] performing query request analysis and routing distribution based on the distributed query routing table, and triggering query plan adjustment when query response time exceeds a preset time threshold, and returning data results obtained from the central database according to the adjusted query plan to a requestor.

[0011] constructing field dependency relationship of a data table as a directed acyclic graph, dividing nodes into real-time check nodes and delayed check nodes based on data integrity requirements, performing data consistency verification on the real-time check nodes, performing asynchronous consistency verification on the delayed check nodes, triggering data correction and re-performing consistency verification when detecting that node data is inconsistent, pushing transaction identifiers carrying timestamps after consistency verification, the transaction identifiers using unique identification codes to mark the data change messages, performing priority identification on concurrent data change operations based on the transaction identifiers in combination with scheduling of a token bucket algorithm, and storing the data change operations that pass verification and the corresponding transaction identifiers to a central database;

[0012] constructing field dependency relationship of a data table as a directed acyclic graph, the directed acyclic graph including a field node set and a dependency relationship edge set, calculating node importance scores based on out-degree, centrality and business importance of field nodes, and dividing field nodes into real-time check nodes and delayed check nodes according to the node importance scores;

[0013] performing local consistency verification and global consistency verification on the real-time check nodes, the local consistency verification being determined based on deviation rate of a current value of a node from an expected value, and the global consistency verification being determined based on deviation of the current value of the node from a weighted derived value of all in-degree neighbor nodes;

[0014] performing batch asynchronous verification on the delayed check nodes according to verification time intervals, and dividing delayed check nodes whose time difference between a last verification time and a current time exceeds a preset time interval into a same batch for verification;

[0015] When detecting that the target node data is inconsistent, a set of upstream nodes directly affecting the target node is obtained, a correction amount required for adjustment of the upstream nodes and the target node is respectively calculated, and a correction strategy is determined according to the correction amount;

[0016] A set of affected nodes is determined based on the dependency relationship of the directed acyclic graph, consistency verification is performed again on the nodes in the set of affected nodes in accordance with a dependency topological order, and a node state marker is updated.

[0017] The step of obtaining a set of upstream nodes directly affecting the target node, calculating a correction amount required for adjustment of the upstream nodes and the target node, and determining a correction strategy based on the correction amount comprises:

[0018] A historical correction record of the target node is obtained, a historical correction frequency and a correction amplitude of each upstream node are calculated, a business weight of each node in the set of upstream nodes is obtained, the business weight is determined based on a business attribute of the node, the business weight is combined with the historical correction frequency and the correction amplitude, a correction cost of each upstream node is calculated, the upstream nodes are sorted according to the values of the correction cost, and an upstream node with the minimum correction cost is selected for correction, and a correction amount of the upstream node is calculated based on a current value of the selected upstream node and a dependency relationship of the target node, in combination with a mean value of the historical correction amplitude;

[0019] A deviation between a real-time data value and a historical stable value when the upstream node correction operation is executed is obtained, and a correction amount of the target node is obtained through weighted calculation; when the correction amount of the target node exceeds a preset correction amount threshold, the correction operation is split into a plurality of sub-correction steps, and a correction amount of each sub-correction step does not exceed the preset correction amount threshold;

[0020] Before the correction operation is performed, a correction amplitude is controlled based on the number of affected nodes; during the correction process, a correction log is maintained, and a compensation operation is performed when an exception is detected.

[0021] After the consistency verification, a transaction identifier carrying a timestamp is pushed, and the step of marking the data change message by a unique identifier code comprises:

[0022] A current timestamp and a source identifier of the data change message are obtained; a unique identifier code is generated based on a UUID algorithm, the timestamp and the source identifier are combined and encoded to obtain the transaction identifier; and the transaction identifier is associated to the data change message in the form of a key-value pair;

[0023] A transaction identifier index table is constructed, the transaction identifier index table comprises a transaction identifier as a primary key, an associated message for tracking a data source, a generation time for time sequence processing, and a message state for process control; when the data change message completes the consistency verification, the message state is updated to a verified state.

[0024] The step of priority identification and scheduling of concurrent data change operations based on the transaction identification in combination with the token bucket algorithm comprises:

[0025] An aging property, a spatial locality property, and a data dependency degree property of a target data change operation are obtained, a time difference between the aging property and a deadline is calculated to obtain a time urgency, a resource utilization rate is calculated based on the spatial locality property and a system resource utilization rate to obtain a resource occupation degree, and a dependency chain information is calculated based on the data dependency degree property to obtain a dependency chain complexity; the time urgency, the resource occupation degree, and the dependency chain complexity are combined to obtain a scheduling priority;

[0026] A multi-level token bucket corresponding to the scheduling priority is constructed, a token generation rate is dynamically adjusted based on a system load level, remaining tokens of a high-priority token bucket are cascaded and distributed according to a request proportion of a low-priority token bucket, and the target data change operation is selected based on the number of tokens in each level of the token bucket to form a set of to-be-executed operations;

[0027] During execution of the set of to-be-executed operations, a deadlock is detected, when a deadlock is detected, a maximum value between an original priority and a priority accumulated due to blocking is set as a temporary priority for each data change operation in the deadlock chain, a compensation cost is calculated, and when the compensation cost is less than a preset cost threshold, the set of to-be-executed operations is re-executed according to an execution order adjusted based on the temporary priority.

[0028] A Bloom filter is used to collect data access hotspot information of the central database, and a distributed query routing table is generated based on dynamic adjustment of data sharding and cache distribution according to the data access hotspot information.

[0029] A counting Bloom filter is used to collect data access information of the central database, a count value of the data access information is updated by decay according to an interval from a current time, and a data access hotspot level is obtained by logarithmic operation of the updated count value and an average count value of the data access information.

[0030] A load balancing degree of data sharding is obtained based on weighted combination of a sharding storage occupation proportion, a sharding hotspot quantity proportion, and a sharding query quantity proportion, and a splitting or merging operation of the sharding is triggered according to a size relationship between the load balancing degree and a preset balancing degree threshold.

[0031] Cache space is allocated among a plurality of cache layers according to the data access hotspot level, a size of the cache space is proportional to an exponential value of the data access hotspot level, and a cache item score is obtained by weighted calculation of an access frequency, a recent access time, and an access overhead of cache data.

[0032] The data access hotspot level, the load balancing degree and the cache item score are taken as routing features to construct a distributed query routing table, a decision weight of each routing item is calculated, the decision weight is obtained by multiplying the ratio of the hotspot level of the node corresponding to the routing item to the hotspot levels of all nodes, the load balancing degree of the node and the cache item score of the node, and a path with the highest decision weight is selected as a query routing path.

[0033] Based on the distributed query routing table, a query request is parsed and routed and distributed, and when a query response time exceeds a preset time threshold, a query plan adjustment is triggered, and data results are obtained from the central database according to the adjusted query plan and returned to a requester, and the steps include:

[0034] A query request is parsed to obtain a query field set and a query condition set, and routing items matched with the query field set and the query condition set are obtained from the distributed query routing table to form a candidate routing item set;

[0035] A routing score of each routing item in the candidate routing item set is calculated, the routing score is obtained by combining a data coverage ratio, a load balancing degree and a network transmission delay time of the node corresponding to the routing item, and a node corresponding to a routing item with the highest routing score is selected as a query execution node;

[0036] A time consumption value in a query execution process is obtained, when the time consumption value exceeds a preset time threshold, a candidate optimization scheme set is generated based on execution performance statistical information, the candidate optimization scheme set includes a parallel number adjustment scheme, a node reallocation scheme, a calculation pushdown scheme and an approximate calculation scheme, an execution cost of each optimization scheme is calculated based on a combination of resource consumption, network transmission and execution time estimation value;

[0037] An optimization scheme with the minimum execution cost is selected as an optimized query plan, and a query result is obtained by execution, performance statistical information of the query execution node is stored in the distributed query routing table, and the query result is returned to the requester.

[0038] In a second aspect of the embodiment of the application, a data decoupling centralized query system triggered by an MQ message is provided, and the system includes:

[0039] A first unit is configured to receive a data change message from a message queue;

[0040] The second unit is configured to construct a field dependency relationship of a data table as a directed acyclic graph, divide nodes into real-time check nodes and delayed check nodes based on data integrity requirements, perform data consistency verification on the real-time check nodes, perform asynchronous consistency verification on the delayed check nodes, trigger data correction and re-perform consistency verification when detecting inconsistent node data, push a transaction identifier carrying a timestamp after consistency verification, the transaction identifier uses a unique identifier code to mark the data change message, perform priority identification on concurrent data change operations based on the transaction identifier and combine a token bucket algorithm for scheduling, and store the verified and scheduled data change operations and corresponding transaction identifiers to a central database.

[0041] The third unit is configured to collect data access hotspot information of the central database using a Bloom filter, dynamically adjust data sharding and cache distribution according to the data access hotspot information, and generate a distributed query routing table.

[0042] The fourth unit is configured to parse and route distribute a query request based on the distributed query routing table, trigger query plan adjustment when a query response time exceeds a preset time threshold, and return data results obtained from the central database according to the adjusted query plan to a requester.

[0043] In a third aspect, an electronic device is provided, including:

[0044] a processor;

[0045] a memory for storing processor-executable instructions;

[0046] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0047] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0048] The method provided by the application can balance the system processing efficiency and the data quality guarantee demand, and avoid the system performance bottleneck problem caused by full real-time check in the traditional method.

[0049] The application introduces a scheduling mechanism combining transaction identification and token bucket algorithm, performs priority management on concurrent data change operations, solves the problem of data change conflict in a high concurrency scenario, collects data access hotspot information through a Bloom filter, dynamically adjusts data sharding and cache distribution, improves the response speed of data query, and reduces system resource consumption.

[0050] The application realizes intelligent routing and adaptive adjustment of query requests, can automatically trigger query plan optimization when the query response time exceeds a threshold, thereby guaranteeing the query efficiency in a large-scale data processing environment, and provides technical support for efficient data access in a distributed data environment. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of a data decoupling centralized query method based on MQ message triggering of the embodiment of the application is shown in

[0052] Figure 2 A priority and token bucket algorithm scheduling flowchart based on transaction identification. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0054] The technical solutions of the application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0055] Figure 1 A flowchart of a data decoupling centralized query method based on MQ message triggering of the embodiment of the application is shown in Figure 1 The method comprises the following steps.

[0056] Receiving a data change message from a message queue;

[0057] The field dependency relationship of a data table is constructed as a directed acyclic graph, nodes are divided into real-time check nodes and delayed check nodes based on data integrity requirements, data consistency verification is performed on the real-time check nodes, asynchronous consistency verification is performed on the delayed check nodes, when inconsistent node data is detected, data correction is triggered and consistency verification is performed again, after consistency verification, a transaction identifier carrying a timestamp is pushed, the transaction identifier uses a unique identifier to mark the data change message, based on the transaction identifier, concurrent data change operations are prioritized and token bucket algorithm scheduling is combined, data change operations that pass verification and scheduling and corresponding transaction identifiers are stored in a central database;

[0058] A Bloom filter is used to collect data access hotspot information of the central database, data shards and cache distribution are dynamically adjusted according to the data access hotspot information, and a distributed query routing table is generated;

[0059] Based on the distributed query routing table, a query request is parsed and routed and distributed, when the query response time exceeds a preset time threshold, query plan adjustment is triggered, and data results are obtained from the central database according to the adjusted query plan and returned to the requester.

[0060] In an optional implementation, the step of constructing the field dependency relationship of a data table as a directed acyclic graph, dividing nodes into real-time check nodes and delayed check nodes based on data integrity requirements, performing data consistency verification on the real-time check nodes, and performing asynchronous consistency verification on the delayed check nodes, when inconsistent node data is detected, triggering data correction and re-performing consistency verification, includes:

[0061] The field dependency relationship of a data table is constructed as a directed acyclic graph, the directed acyclic graph includes a field node set and a dependency relationship edge set, node importance scores are calculated based on the out-degree, centrality and business importance of field nodes, and field nodes are divided into real-time check nodes and delayed check nodes according to the node importance scores;

[0062] Local consistency verification and global consistency verification are performed on the real-time check nodes, the local consistency verification is determined based on the deviation rate of the current value of the node and the expected value, and the global consistency verification is determined based on the deviation of the current value of the node and the weighted derived value of all in-degree neighbor nodes;

[0063] The delayed check nodes are batched and asynchronously checked according to verification time intervals, and the delayed check nodes whose time difference between the last verification time and the current time exceeds a preset time interval are divided into the same batch for verification;

[0064] When detecting that the target node data is inconsistent, a set of upstream nodes directly affecting the target node is obtained, a correction amount required for adjustment of the upstream nodes and the target node is respectively calculated, and a correction strategy is determined according to the correction amount;

[0065] A set of affected nodes is determined based on the dependency relationship of the directed acyclic graph, consistency verification is performed again on the nodes in the set of affected nodes in a topological order, and a node state flag is updated.

[0066] For example, the field dependency relationship of a data table is constructed into a directed acyclic graph, which is composed of a set of field nodes and a set of dependency relationship edges. For example, in a customer order scenario, the order total price field depends on the commodity unit price and the purchase quantity field, and this dependency relationship can be represented as a directed edge from the commodity unit price and the purchase quantity to the order total price. By scanning the constraint relationship, trigger, and calculation logic in the application layer code of the data table, the dependency relationship between the fields is automatically extracted, and a complete directed acyclic graph is constructed.

[0067] After the graph is constructed, the importance score of each field node considers three factors of the out-degree, the centrality, and the business importance of the node. The out-degree represents how many other fields depend on the field, for example, the user account balance field is referenced by multiple business scenarios, and its out-degree is high. The centrality measures the importance of the position of the node in the entire graph, which is calculated by the shortest path between nodes. The business importance is evaluated by domain experts according to the business value, for example, the order payment status field has a high business importance in transactions. The three factors are respectively assigned weight values, such as the out-degree weight 0.3, the centrality weight 0.4, and the business importance weight 0.3, and the final node importance score is obtained by weighted summation. The nodes with a score exceeding a preset threshold (such as 0.7) are divided into real-time verification nodes, which need to be verified immediately when the data changes; the nodes with a score below the threshold are divided into delayed verification nodes, which can be verified asynchronously when the load is low. For example, the order payment status field is divided into a real-time verification node due to its high importance, and the order note field is a delayed verification node due to its low importance.

[0068] For real-time verification nodes, local consistency verification and global consistency verification are performed. The local consistency verification compares the deviation rate of the current value of the node and the expected value. For example, when a user completes an order payment of 100 yuan, the expected account balance decreases by 100 yuan, and if the actual amount of decrease deviates from the expected value by more than the allowed range (such as 1%), it is determined to be inconsistent. The global consistency verification calculates a weighted derivation value based on all the in-degree neighbor nodes of the node, and compares it with the current value of the node. For example, the current value of the commodity inventory field should be equal to the initial inventory minus all the sold quantities, and the derivation value is calculated according to this relationship, which is compared with the current inventory value. If the deviation exceeds the threshold, it is determined to be inconsistent.

[0069] For the delayed check nodes, asynchronous checking is performed in batches according to the verification time interval. The last verification time of each delayed check node is recorded, and when the difference between the current time and the last verification time exceeds the preset interval (such as 12 hours), the node is added to the verification queue. Multiple delayed check nodes that expire within the same time period are divided into the same batch, and are uniformly checked when resources are sufficient. For example, the non-core configuration information field is verified every 24 hours, and the user preference setting field is verified every 12 hours.

[0070] When detecting that the target node data is inconsistent, the correction strategy needs to be determined. First, the upstream node set directly affecting the target node is obtained, that is, all nodes directly pointing to the target node in the directed graph. For example, when the order total price is inconsistent, the commodity unit price and purchase quantity are identified as upstream nodes. Based on the historical correction record, the correction frequency and correction amplitude of each upstream node are analyzed, and the nodes with high frequency and large amplitude are usually stable. At the same time, the business weight of each upstream node is evaluated, and the data sensitivity (such as high payment information sensitivity), business importance (such as high importance of core transaction data), and historical stability (such as low error rate of the field) are comprehensively considered. The business weight and historical correction index are combined to calculate the correction cost of each upstream node. For example, the cost of modifying sensitive payment information is high, while the cost of adjusting non-sensitive note information is low. The upstream node with the minimum correction cost is selected for correction.

[0071] For the selected upstream node, the correction amount is calculated, and the correction amount of the target node is derived based on the dependency relationship. For example, after the commodity unit price field is corrected, the order total price field needs to be adjusted accordingly. When the correction amount of the target node exceeds the preset threshold (such as an account balance adjustment exceeding 500 yuan), the correction operation is split into multiple sub-steps, and the correction amount of each step is controlled within the threshold to reduce the risk. Before performing the correction, the affected node range is evaluated, and when the impact range is too large (such as more than 50 nodes), the single correction amplitude is reduced by a preset proportion (such as reduced to 50% of the original plan), and the correction is completed in multiple times.

[0072] Detailed correction logs are maintained in each correction operation, recording the data value before correction, the data value after correction, and the correction timestamp. When an exception is detected during the correction process, such as performance degradation or error, a compensation operation is performed according to the correction log in reverse order of the timestamp to restore to the state before correction. After completing the correction, based on the dependency relationship of the directed acyclic graph, the set of all affected nodes is determined, and the consistency of these nodes is re-verified according to the topology of the dependency and the state of the nodes is updated.

[0073] The application realizes accurate checking and intelligent correction of data consistency by constructing a data-dependent directed acyclic graph, significantly improving data quality and reliability. The real-time checking and delayed checking strategies optimize resource utilization efficiency and reduce peak load. The correction strategy based on historical data and business weight reduces unnecessary data modification and improves the accuracy and security of correction operations. The correction log and compensation mechanism enhance fault tolerance and recoverability, meeting the strict requirements of enterprise-level applications for data consistency.

[0074] In an optional embodiment, a set of upstream nodes directly affecting the target node is obtained, and the correction amount required for adjustment of each upstream node and the target node is calculated respectively. The step of determining a correction strategy according to the correction amount comprises:

[0075] The historical correction records of the target node are obtained, and the historical correction frequency and correction amplitude of each upstream node are calculated. The business weight of each node in the set of upstream nodes is obtained, and the business weight is determined based on the business attributes of the node. The business weight, historical correction frequency and correction amplitude are combined to calculate the correction cost of each upstream node. The upstream nodes are sorted according to the numerical value of the correction cost, and the upstream node with the minimum correction cost is selected for correction. The correction amount of the upstream node is calculated based on the current value of the selected upstream node and the dependency relationship of the target node, combined with the average value of the historical correction amplitude.

[0076] The deviation between the real-time data value and the historical stable value of the upstream node when the correction operation is executed is obtained, and the correction amount of the target node is calculated by weighted calculation. When the correction amount of the target node exceeds the preset correction amount threshold, the correction operation is split into multiple sub-correction steps, and the correction amount of each sub-correction step does not exceed the preset correction amount threshold.

[0077] Before executing the correction operation, the correction amplitude is controlled based on the number of affected nodes. During the correction process, a correction log is maintained, and a compensation operation is performed when an abnormality is detected.

[0078] For example, when the data inconsistency of the target node is detected, the upstream nodes causing the inconsistency need to be accurately identified and the appropriate correction amount needs to be calculated. In the order processing of an e-commerce platform, if the total order amount does not match the actual calculated amount, all upstream nodes directly affecting the total order amount, including the unit price of goods, purchase quantity, discount, etc., can be identified through the dependency relationship graph. The breadth-first search algorithm is used to traverse the directed acyclic graph to find all nodes directly pointing to the target node, forming a set of upstream nodes. For example, for the total order amount node, three upstream nodes of the unit price of goods, quantity and discount are identified.

[0079] After determining the set of upstream nodes, the historical correction records of the target node are obtained. These records contain information such as the time of each correction, the value before correction, and the value after correction. Statistical analysis of the correction of each upstream node is performed from the historical records to calculate the correction frequency and the correction amplitude. The correction frequency represents the number of times the node has been corrected in the past period of time, for example, the commodity unit price node has been corrected 5 times in the past 30 days, with a correction frequency of 0.17 times / day. The correction amplitude refers to the range of value change of each correction, such as the average correction amplitude of commodity unit price is 3% of the original value. High correction frequency and large correction amplitude usually indicate that the node data stability is poor and is not suitable as the preferred target for correction.

[0080] At the same time, the business weight of each node in the set of upstream nodes is evaluated. The business weight is determined based on the business attributes of the node, including factors such as data sensitivity, business criticality, and compliance requirements. In the order, the payment amount node has high sensitivity due to the involvement of financial data, and is given a higher weight such as 0.9; the commodity description node has low sensitivity, and is given a lower weight such as 0.2. The weight values of various nodes can be determined through a pre-set business rule table, which is maintained by domain experts according to business characteristics.

[0081] The business weight is combined with the historical correction indicators to calculate the correction cost of each upstream node. The correction cost reflects the risk and cost of modifying the node, which is derived from the weighted combination of business weight, correction frequency, and correction amplitude. For example, for the payment amount node, even if the correction frequency is low, its correction cost is still high due to the high business weight. When calculating the correction cost, the business weight is multiplied by the weighted sum of the correction frequency and the correction amplitude, such as the value of the correction cost = business weight × (0.4 × correction frequency + 0.6 × correction amplitude). All upstream nodes are sorted in ascending order of correction cost, and the node with the smallest correction cost is selected as the priority correction object.

[0082] After determining the upstream node to be corrected, the correction amount of the upstream node is calculated based on the current value of the node and the dependency relationship with the target node, combined with the mean value of the historical correction amplitude. In the order amount calculation, if the commodity unit price is selected for correction, the relationship between the unit price and the total amount (total amount = unit price × quantity - discount) is analyzed, and the average amplitude of historical unit price correction is referred to, such as the past correction amplitude mean value is 3%, then the correction amplitude of this time is also controlled within a similar range. The correction amount calculation needs to consider business constraints, such as the commodity unit price cannot be negative.

[0083] The deviation of the real-time data value and the historical stable value during the execution of the upstream node correction operation is obtained, and the correction amount of the target node is determined by weighted calculation. The historical stable value can be obtained by calculating the average value or the median value of the node in the past period of time. For example, the historical stable value of the unit price of a certain commodity is 100 yuan, and the current value is 150 yuan, with a deviation of 50%. Multiply this deviation by the dependency weight to calculate the correction amount of the target node. For example, the total order amount needs to be reduced by a certain percentage. When the calculated correction amount of the target node exceeds a preset threshold (such as 5%), the correction operation is split into multiple sub-steps to gradually achieve the target correction amount. For example, a 10% correction amount is split into two 5% correction operations, and the stability is verified after each correction to ensure safety.

[0084] Before performing the correction, the number of affected nodes is evaluated, and the correction amplitude is controlled based on the number. Through a graph traversal algorithm, all downstream nodes affected by the correction are identified from the corrected node. If the number of affected nodes exceeds a preset threshold (such as 50 nodes), the single correction amplitude is reduced by a predetermined percentage. For example, the original plan is to correct the unit price of a commodity by 5%, but when the number of affected nodes is too large, it is reduced to 2.5% and the correction target is completed in multiple times. The execution time of the correction is also planned according to the importance of the nodes, and the correction with a large impact range is executed during a period with low load.

[0085] A detailed correction log is maintained during each correction process, recording the data value before correction, the data value after correction, the correction timestamp, and the associated transaction identifier for each modified node. For example, the unit price of a commodity is changed from 100 yuan to 95 yuan, the timestamp is 2023-05-2015:30:45, and the associated order number is ORDER12345. When an abnormal situation is detected during the correction process, such as a service response time exceeding a threshold, a transaction failure rate increasing, or a resource occupation being too high, the current correction operation is immediately suspended, and a compensation operation is performed based on the correction log. The compensation operation restores the modified nodes to their pre-correction state in reverse order of the timestamp to ensure consistency and business continuity.

[0086] The present application realizes accurate positioning and intelligent correction of data inconsistency problems by comprehensively analyzing the business attributes, historical correction situations and dependency relationships of nodes; the step-by-step correction and dynamic adjustment mechanism effectively controls the correction impact range; the perfect log recording and compensation mechanism enhances the fault tolerance and recoverability, providing a comprehensive solution for data consistency management.

[0087] In an optional implementation, after the consistency check, the transaction identifier carrying the timestamp is pushed, and the step of marking the data change message with a unique identifier code using the transaction identifier includes:

[0088] Obtain the current timestamp and the source identifier of the data change message; generate a unique identifier code based on the UUID algorithm, combine the timestamp and the source identifier to obtain the transaction identifier; associate the transaction identifier to the data change message in the form of key-value pair;

[0089] Build a transaction identifier index table, which includes the transaction identifier as the primary key, the associated message for tracking the data source, the generation time for time sequence processing, and the message state for flow control; update the message state to the checked state when the data change message completes the consistency check.

[0090] Illustratively, the transaction identifier is a key element to ensure data consistency and traceability. When the consistency check of the data change message is completed, a unique transaction identifier needs to be assigned to the message and pushed to the downstream processing link.

[0091] Obtain the current timestamp and the source identifier of the data change message. The timestamp adopts millisecond-level precision to ensure the time sequence accuracy in high concurrency scenarios, and the format is "yyyyMMddHHmmssSSS", such as "20230615142536789". The source identifier is used to indicate the initiator of the data change, which is the application service name, database instance identifier, or user session identifier, etc. In order processing, the source identifier is "order-service" or "inventory-update", etc. The source identifier is extracted from the request header or message metadata, and if it does not exist, the default value "system" is used.

[0092] A 128-bit random identifier is generated by using version 4 UUID algorithm, such as "550e8400-e29b-41d4-a716-446655440000". The advantage of UUID is that it does not rely on centralized identifier allocation mechanism, and it is suitable for distributed environment. In order to improve the readability and convenience of using UUID, it will be converted to a 32-bit hexadecimal string without separator, such as "550e8400e29b41d4a716446655440000".

[0093] The acquired timestamp is combined with the source identifier to generate the final transaction identifier. The combination method is to place the timestamp in the prefix position, the source identifier is hashed and used as the middle part, and the UUID is used as the suffix. For example, the timestamp "20230615142536789", the hash value of the source identifier "order-service" is "os7d", and the UUID is simplified to the first 8 bits "550e8400". The final combined transaction identifier is "20230615142536789-os7d-550e8400". This combination method ensures the uniqueness of the identifier and makes the identifier itself carry time and source information, which is convenient for troubleshooting and tracking.

[0094] After generating the transaction identifier, it is associated with the data change message in the form of key-value pairs. In the message queue, the transaction identifier is added to the properties field of the message, with the key name "transaction_id" and the value of the generated transaction identifier. For example, the properties field of an order update message contains {"transaction_id":"20230615142536789-os7d-550e8400","priority":"high","retry_count":"0"}. For database operations, the transaction identifier is recorded in the operation log table or added as a metadata field to the related data record.

[0095] Build a transaction identifier index table as a central storage structure for transaction processing and tracking. The index table is implemented in a relational database and contains four core fields: transaction identifier, associated message, generation time, and message status. The transaction identifier field is set as the primary key, with the type VARCHAR(50) to store the complete transaction identifier string. The associated message field is of type TEXT, storing the JSON serialized representation of the original data change message, which is convenient for tracing data sources and content. For example, an order update message is stored as {"order_id":"ORD12345","update_type":"status_change","new_status":"paid","old_status":"pending"}. The generation time field is of type TIMESTAMP, recording the exact time point when the transaction identifier is generated, supporting time sequence queries and processing. The message status field is of type VARCHAR(20), used to mark the processing status of the message, with the initial value set to "pending" indicating the pending state.

[0096] When the data change message completes the consistency check, the message status of the corresponding record in the index table is updated. The message status of the check-through is updated to "verified", indicating that it has passed the consistency check. The state update operation is implemented through a SQL statement, such as "UPDATE transaction_index SET message_status ='verified' WHERE transaction_id = '20230615142536789-os7d-550e8400'". After the state update is completed, the message carrying the transaction identifier is pushed to the downstream processing link, such as a data synchronization service or a business processing service. The downstream service can query the index table through the transaction identifier to obtain the complete transaction context information, ensuring the continuity and consistency of processing.

[0097] The combined encoding mode of the transaction identifier of the application provides intuitive timing and source information, facilitating problem positioning and data auditing; the design of the index table supports efficient transaction query and state update, enhancing observability and manageability.

[0098] In an optional implementation, the step of performing priority identification on concurrent data change operations based on the transaction identifier and scheduling in combination with a token bucket algorithm includes:

[0099] An aging property, a spatial locality property, and a data dependency degree property of the target data change operation are obtained; a time emergency degree is calculated by calculating a time difference between the aging property and a deadline, a resource occupation degree is calculated by calculating a system resource utilization rate based on the spatial locality property, and a dependency chain complexity is calculated by calculating dependency chain information based on the data dependency degree property; the time emergency degree, the resource occupation degree, and the dependency chain complexity are combined by weighting to obtain a scheduling priority;

[0100] A multi-level token bucket corresponding to the scheduling priority is constructed; a token generation rate is dynamically adjusted based on a system load level; remaining tokens of a high-priority token bucket are cascaded and distributed according to a request proportion of a low-priority token bucket; and the target data change operation is selected based on the number of tokens in each level of the token bucket to form a set of to-be-executed operations;

[0101] During execution of the set of to-be-executed operations, a deadlock is detected; when a deadlock is detected, a maximum value between an original priority and a priority accumulated due to blocking is set as a temporary priority for each data change operation on the deadlock chain; a compensation cost is calculated; and when the compensation cost is less than a preset cost threshold, the set of to-be-executed operations is re-executed in an execution order adjusted based on the temporary priority.

[0102] In combination Figure 2The priority based on the transaction identifier and the token bucket algorithm scheduling flowchart are described, and three core attributes of the target data change operation are obtained: timeliness attribute, spatial locality attribute and data dependency attribute. The timeliness attribute represents the urgency of the operation, which is usually extracted from the message header, such as priority mark or deadline mark. For example, the inventory deduction operation is marked as "high_priority" and the deadline is set to 300 milliseconds after the current time. The spatial locality attribute describes the data distribution characteristics involved in the operation, including the storage node where the data is located, the access mode and the data size. For example, the order update operation involves the order master table, the commodity table and the user table distributed in multiple nodes, the data access mode is "read more and write less", and the total data size is 2KB. The data dependency attribute identifies the dependency relationship between the operation and other operations, which is represented by a directed acyclic graph. For example, the order payment confirmation operation depends on the account balance check operation and the inventory confirmation operation.

[0103] The time difference between the timeliness attribute and the deadline is calculated to obtain the time urgency, and the time difference between the current time and the deadline of the operation is calculated (millisecond level precision). Then, a time urgency conversion function is applied to map the time difference to a standardized urgency score, with a value range of 0 to 1. The conversion function adopts exponential decay: urgency = 1-(time difference / maximum tolerance time difference) 2where the maximum tolerated time difference is a preset constant, such as 1000 milliseconds; the time urgency of an operation with a deadline 50 milliseconds later than the current time is calculated as 0.9; and the time urgency of an operation with a deadline 500 milliseconds later than the current time is calculated as 0.5. The resource occupation degree is calculated according to the spatial locality property, and a weighted summation method is used to comprehensively evaluate the CPU, memory, network, and storage resource requirements. Resource occupation degree = (CPU weight x CPU requirement) + (memory weight x memory requirement) + (network weight x network requirement) + (storage weight x storage requirement). The resource requirements are obtained by looking up the operation type table to obtain standardized values, and the weights are dynamically adjusted according to the current bottleneck resource. For example, when the CPU utilization is 70%, the CPU weight is set to 0.5, and the weights of other resources are each 0.16. For a CPU-intensive operation (CPU requirement 0.8, other requirements 0.3), the calculated resource occupation degree is 0.8 x 0.5 + 0.3 x 0.16 x 3 = 0.544. Dependency chain complexity = dependency operation number factor + dependency chain length factor + critical path factor. Dependency operation number factor = number of dependent operations / maximum number of dependent operations (preset as 10); dependency chain length factor = longest dependency chain length / maximum chain length (preset as 5); and critical path factor is a preset value according to whether the operation is on the critical path (0.3 on the critical path, 0 off the critical path). For a critical path operation that depends on three preceding operations and forms a two-level dependency relationship, the dependency chain complexity is calculated as: 3 / 10 + 2 / 5 + 0.3 = 0.3 + 0.4 + 0.3 = 0.7. For an independent operation without dependency, the three factors are all 0, and the complexity is 0.

[0104] The time urgency, resource occupation degree, and dependency chain complexity are combined by weighting to obtain the final scheduling priority. The weighting coefficients are dynamically adjusted based on the current state, with the weight of the resource occupation degree increased in high-load situations and the weight of the time urgency increased in low-latency requirement scenarios. For example, under normal load, the weights of the three are 0.4, 0.3, and 0.3, respectively; and when resources are scarce, the weights are adjusted to 0.3, 0.5, and 0.2, respectively. Assuming that the time urgency of an operation is 0.9, the resource occupation degree is 0.6, and the dependency chain complexity is 0.4, and the weights are 0.4, 0.3, and 0.3, respectively, the calculated priority is 0.66. The calculated priority value is mapped to a predefined priority level, such as "critical" (0.8-1.0), "high" (0.6-0.8), "medium" (0.4-0.6), "low" (0.2-0.4), and "background" (0-0.2).

[0105] Based on the priority level, a multi-level token bucket structure is constructed, with each priority corresponding to an independent token bucket. Each token bucket contains parameters such as the current number of available tokens, maximum token capacity, token generation rate, and last update time. For example, the "critical" level token bucket is configured with a maximum capacity of 100 tokens and a generation rate of 50 tokens per second, while the "background" level is configured with a maximum capacity of 500 tokens and a generation rate of 20 tokens per second. The current load level is continuously monitored, including CPU usage, memory occupancy, network throughput, and I / O latency, and the token generation rate is dynamically adjusted. When the load exceeds the warning threshold (e.g., CPU utilization > 80%), the token generation rate is reduced by a preset proportion, and the token supply for high-priority buckets is prioritized. For example, in high-load situations, the "critical" level generation rate remains unchanged, while the generation rates of other levels are reduced by 30%.

[0106] When there are fewer high-priority operations and the token bucket accumulates surplus tokens, the surplus tokens will be allocated to low-priority buckets according to the request proportion of low-priority token buckets. For example, there are 30 surplus tokens in the "critical" level, and the request proportion of "high" and "medium" levels is 2:1, so 20 tokens are allocated to the "high" level and 10 tokens are allocated to the "medium" level. This mechanism not only guarantees resource guarantee for high-priority operations, but also improves overall resource utilization efficiency. Based on the number of tokens in each level of token bucket, the target data change operation is selected to form a set of operations to be executed. The operation selection follows the principles of priority first and first-come-first-served, while considering the dependency relationship between operations. Starting from the highest priority token bucket, operations are selected in the order of operation arrival until the token bucket is empty or there are no operations that meet the conditions. For example, when there are 50 tokens in the "critical" level and 30 operations in the queue, all 30 operations are included in the set of operations to be executed, and operations are selected from the "high" level.

[0107] In the process of executing the set of operations to be executed, potential deadlock situations are continuously detected. The deadlock detection is based on a resource allocation graph algorithm, periodically scanning the resource request and holding relationship between operations to identify loop structures. When a deadlock is detected, the priority of each data change operation on the deadlock chain is re-evaluated. The maximum value of the original priority and the priority accumulated due to blocking is set as the temporary priority. The blocking accumulated priority is proportional to the time the operation is blocked, the longer the blocking time, the higher the accumulated priority. For example, an operation with an original priority of 0.6, if blocked for 200 milliseconds, the accumulated priority increases to 0.75, and the temporary priority takes 0.75. The compensation cost of deadlock resolution is calculated, including the cost of rolling back the executed operations and the time loss of rescheduling. When the compensation cost is less than the preset cost threshold, the set of operations to be executed is re-executed according to the execution order adjusted by the temporary priority. If the compensation cost is too high, some low-priority operations are sacrificed and the deadlock is directly resolved.

[0108] The operation scheduled by the checksum and its transaction identifier are stored to the central relational database through the transaction log table, realizing data persistence and tracking management.

[0109] The application realizes fine management of data change operations and efficient allocation of resources; multi-dimensional priority calculation ensures timely processing of key operations, the multi-level structure of the token bucket and the cascading allocation mechanism improve resource utilization efficiency; dynamically adjusted token generation rate and deadlock handling strategy enhance adaptability and stability, providing a solid guarantee for data consistency management in a high-concurrency environment.

[0110] In an optional embodiment, a Bloom filter is used to collect data access hotspot information of the central database, and the step of generating a distributed query routing table according to the data access hotspot information includes:

[0111] A counting Bloom filter is used to collect data access information of the central database, the count value of the data access information is updated by decay according to the interval from the current time, and the updated count value and the average count value of the data access information are logarithmically operated to obtain a data access hotspot level;

[0112] The load balancing degree of the data shard is obtained based on the weighted combination of the shard storage occupancy ratio, the shard hotspot quantity ratio and the shard query quantity ratio, and the splitting or merging operation of the shard is triggered according to the size relationship between the load balancing degree and the preset balancing degree threshold;

[0113] The cache space is allocated among the multiple cache layers according to the data access hotspot level, the size of the cache space is proportional to the exponential value of the data access hotspot level, and the access times, the latest access time and the access overhead of the cache data are weighted to obtain a cache item score;

[0114] The data access hotspot level, the load balancing degree, and the cache item score are used as routing features to construct a distributed query routing table, a decision weight of each routing item is calculated by multiplying the ratio of the hotspot level of the node corresponding to the routing item to the hotspot levels of all nodes, the load balancing degree of the node, and the cache item score of the node, and a path with the highest decision weight is selected as a query routing path.

[0115] Exemplarily, in the data decoupled centralized query, a Bloom filter is used to efficiently collect data access hotspot information to provide a basis for dynamically adjusting data sharding and cache distribution. A counting Bloom filter is used to collect data access information of a central database, which not only records whether an element exists, but also records the access times of each element. The counting Bloom filter is composed of m counters and k independent hash functions. Each data item is mapped to k counter positions through k hash functions. When the data is accessed, the corresponding k counter values are increased by 1. In the implementation, the parameter configuration of m=10,000,000 and k=7 is selected to balance the memory occupation and the false positive rate. For the record of order ID "ORD123456" in the order data table, the position set {143928, 5628991, 3489223, 9812734, 2341895, 7890123, 5432167} is calculated using 7 hash functions. Each time the record is accessed, the counters at these positions are increased by 1.

[0116] The counting values of the data access information are periodically updated to decay, so that the counting values reflect the recent access hotness. The decay update adopts a time decay function to adjust the counting values based on the interval from the current time. The decay formula is: new counting value=original counting value x decay factor, and the decay factor=0.5 raised to the power of (time interval / half-life period). The half-life period is set to 24 hours, that is, the access count weight of 24 hours ago is halved. For example, the counting value of the access record 48 hours ago is decayed to 0.25 times the original value. The updated counting value and the average counting value of the data access information are logarithmically operated to obtain the data access hotspot level. The hotspot level calculation formula is: hotspot level=logarithm of (counting value / average counting value) with base 2, plus 1. The average counting value is obtained by averaging all non-zero counter values. For example, if the average counting value is 10 and the counting value of a data item is 80, the hotspot level of the data item is the logarithm of (80 / 10) with base 2, plus 1, which is calculated to be 3.7. The hotspot level is divided into five levels: cold data (≤1), warm data (1-2), hot data (2-3), extremely hot data (3-4), and super hot data (>4).

[0117] The shard storage occupancy ratio represents the unevenness of the usage of the shard storage space. The calculation method is: maximum shard storage occupancy / average shard storage occupancy. The shard hotspot quantity ratio represents the unevenness of the distribution of hotspot data in each shard. The calculation method is: maximum shard hotspot quantity / average shard hotspot quantity. The shard query quantity ratio represents the unevenness of the distribution of query load in each shard. The calculation method is: maximum shard query quantity / average shard query quantity. The weighted sum of the three ratios is obtained, and the weights are 0.3, 0.4, and 0.3, respectively, to obtain the final load balancing degree. The closer the load balancing degree value is to 1, the more balanced it is. For example, a certain shard has 3 shards, the storage occupancy ratio is 1.2, the hotspot quantity ratio is 1.5, and the query quantity ratio is 1.3. The calculated load balancing degree is 1.2 x 0.3 + 1.5 x 0.4 + 1.3 x 0.3 = 1.35.

[0118] According to the size relationship between the load balancing degree and the preset balancing degree threshold, the splitting or merging operation of the shard is triggered. When the load balancing degree exceeds the upper threshold (such as 1.5), the shard splitting operation is triggered. When splitting, the shard with the highest load is selected, and the data of the shard is redistributed to two sub-shards using a consistent hashing algorithm. During the splitting process, the hotspot data is preferentially and evenly distributed to the sub-shards to improve the balancing degree after splitting. When the load balancing degree is lower than the lower threshold (such as 0.7) and the number of shards exceeds the minimum value, the shard merging operation is triggered. When merging, the two shards with the lowest load and adjacent to each other are selected for merging, and the capacity of the merged shard does not exceed the maximum shard capacity limit. For example, when the load balancing degree reaches 1.6, the shard A with the most concentrated hotspots is split into shards A1 and A2, and the hotspot data in the shards is evenly distributed by rehashing, so that the load balancing degree is reduced to 1.3.

[0119] According to the data access hotspot level, the cache space is allocated among multiple cache layers, and a multi-level architecture is adopted, including memory cache, SSD cache and HDD cache. The allocated space of each cache layer is proportional to the exponential value of the data access hotspot level. The allocation ratio calculation formula is: cache space ratio = 2 raised to the power of (hotspot level-1). For example, the cache space ratio of data items with a hotspot level of 4 is 2 raised to the power of (4-1) = 8, indicating that the allocated cache space is 8 times the baseline space. Memory cache is preferentially allocated for super-hot data (>4), SSD cache is allocated for extremely hot data (3-4) and hot data (2-3), HDD cache is allocated for warm data (1-2), and cold data (≤1) does not enter the cache. The cache efficiency is evaluated periodically, and the access times, recent access time and access overhead of the cached data are weighted and calculated to obtain the cache item score. The access times are recorded by a counter, the recent access time is recorded using a timestamp, and the access overhead is estimated according to the average time consumption of data acquisition. The weights of the three are 0.5, 0.3 and 0.2 respectively, and the final score is obtained by weighted summation. For example, a certain cache item has an access time of 100 times (normalized value 0.8), a recent access time of 5 minutes ago (normalized value 0.9), and an access overhead of 10 ms (normalized value 0.6), and its score is 0.8x0.5+0.9x0.3+0.6x0.2=0.79.

[0120] The data access hotspot level, load balancing degree and cache item score are used as routing features to construct a distributed query routing table. The routing table includes four key fields: data item identification, shard location, cache location and decision weight. The decision weight of each routing item is calculated as the basis for query routing decision. The decision weight is obtained by multiplying the hotspot level proportion of the node, the load balancing degree and the cache item score. The hotspot level proportion represents the ratio of the hotspot level of the node to the total of all node hotspot levels. For example, the hotspot levels of three nodes are 3, 2 and 1 respectively, and the hotspot level proportion of the first node is 3 / (3+2+1)=0.5. The hotspot level proportion of a certain routing item corresponding to the node is 0.5, the load balancing degree is 0.9 (the reciprocal is taken, so the smaller the value is, the better), and the cache item score is 0.8, so the decision weight is 0.5x0.9x0.8=0.36. When querying the routing, the path with the highest decision weight is selected as the query routing path. If multiple paths have similar weights (the difference is less than 0.1), the current load of the node will be considered, and the node with lower load will be preferentially selected.

[0121] The application collects data access hotspot information accurately through a Bloom filter, realizes dynamic optimization and adjustment of data sharding and cache distribution, ensures balanced resource utilization through a multi-dimensional load balancing evaluation mechanism, improves overall throughput capacity, optimizes cache hit rate through a multi-level cache allocation strategy based on hotspot levels, reduces query delay, and makes query requests be accurately directed to optimal processing nodes through a routing decision mechanism considering hotspot distribution, load balancing and cache efficiency, thereby significantly improving query efficiency and scalability of the distributed database.

[0122] In an optional implementation, the query request is parsed and routed and distributed based on the distributed query routing table; when the query response time exceeds a preset time threshold, the query plan is adjusted, and the data result is obtained from the central database according to the adjusted query plan and returned to the requester, and the step includes:

[0123] The query request is parsed to obtain a query field set and a query condition set; the routing items matched with the query field set and the query condition set are obtained from the distributed query routing table to form a candidate routing item set;

[0124] The routing score of each routing item in the candidate routing item set is calculated, the routing score is calculated based on the combination of the data coverage ratio, the load balancing degree and the network transmission delay time of the node corresponding to the routing item, and the node corresponding to the routing item with the highest routing score is selected as the query execution node;

[0125] The time consumption value in the query execution process is obtained; when the time consumption value exceeds a preset time threshold, a candidate optimization scheme set is generated based on the execution performance statistical information, including a parallel quantity adjustment scheme, a node redistribution scheme, a calculation pushdown scheme and an approximate calculation scheme; the execution cost of each optimization scheme is calculated based on the combination of resource consumption, network transmission amount and execution time estimation value;

[0126] The optimization scheme with the minimum execution cost is selected as the optimized query plan and executed to obtain the query result; the performance statistical information of the query execution node is stored in the distributed query routing table; and the query result is returned to the requester.

[0127] For example, after receiving a query request, the query request is parsed by a syntax analyzer to obtain a query field set and a query condition set. The query field set contains all data fields that need to be returned, such as "order_id", "product_name", "order_amount", "create_time", etc. when a user queries order information. The query condition set contains all filtering conditions, each of which consists of a field name, an operator, and a comparison value. For example, the condition "order_amount>1000 AND create_time>'2023-01-01'" is parsed into two condition items: {"field":"order_amount","operator":">","value":1000} and {"field":"create_time","operator":">","value":"2023-01-01"}. Compound conditions are decomposed to support AND, OR, NOT, etc. logical operators, and a condition tree structure is constructed to facilitate subsequent matching.

[0128] From the distributed query routing table, a routing item that matches the query field set and the query condition set is obtained to form a candidate routing item set. The matching process adopts a two-stage strategy: field matching and condition matching. Field matching ensures that the routing item can cover all the fields required by the query, and condition matching evaluates whether the routing item meets the index requirements of the query conditions. Routing items with index coverage capability are preferred, i.e. all fields required by the query are included in the index, and no table query is needed. For example, for a query containing "order_id" and "product_name" fields, a joint index routing item containing these two fields is preferred. When a completely matching routing item cannot be found, a partial matching strategy is adopted, and the query is decomposed into multiple sub-queries, which are processed by different routing items. For the query condition "customer_id=12345 AND order_status='paid'", three candidate routing items are found: routing item A contains the customer_id index, routing item B contains the order_status index, and routing item C contains the joint index of both. These three routing items will form the candidate routing item set.

[0129] The routing score is calculated based on a weighted combination of three factors: data coverage ratio, load balancing degree, and network transmission delay time. The data coverage ratio represents the proportion of query fields and conditions that can be directly satisfied by the routing item. The calculation method is: (number of matched fields + number of matched conditions) / (total number of fields + total number of conditions). The load balancing degree reflects the current load status of the node, which is evaluated by CPU usage, memory usage, disk I / O, etc. The value range is 0-1, and the smaller the value, the lower the load. The network transmission delay time is obtained by periodic probing of the node, with a unit of milliseconds. It is converted to a score value between 0 and 1 by normalization, and the smaller the value, the lower the delay. The weights of the three factors are 0.5, 0.3, and 0.2, respectively. The final routing score is obtained by weighted summation. For example, the data coverage ratio of routing item C is 1.0 (complete match), the load balancing degree is 0.4, and the network transmission delay is 15 ms (normalized value is 0.3). The routing score of routing item C is 1.0 x 0.5 + 0.4 x 0.3 + 0.3 x 0.2 = 0.71. The node corresponding to the routing item with the highest routing score is selected as the query execution node. In the above example, if the score of routing item A is 0.52, the score of routing item B is 0.48, and the score of routing item C is 0.71, then the node corresponding to routing item C is selected to execute the query.

[0130] During query execution, time consumption values are obtained, including CPU time, I / O waiting time, and network transmission time, etc. The total execution time is compared with a preset time threshold (such as 200 milliseconds). When the total execution time exceeds the threshold, the query plan adjustment is triggered. Based on the execution performance statistics, multiple candidate optimization schemes are generated, including parallel number adjustment scheme, node redistribution scheme, calculation pushdown scheme, and approximate calculation scheme. The parallel number adjustment scheme improves query efficiency by increasing the number of parallel processing threads, such as increasing the number of 4-thread parallel processing to 8 threads. The node redistribution scheme selects a node with lower load or better data locality to execute the query, such as switching from the master node to the slave node with data replicas. The calculation pushdown scheme pushes the calculation task to the node where the data is located, reducing the data transmission amount, such as pushing the aggregation operation (SUM, AVG, etc.) to the storage node for execution. The approximate calculation scheme provides approximate results by sampling, which is suitable for statistical analysis type queries, such as using percentage sampling instead of full table scan to calculate the average value.

[0131] The execution cost of each optimization scheme is calculated based on the combination of resource consumption, network transmission volume, and execution time estimate. The resource consumption includes the weighted sum of CPU, memory, disk I / O, and other resource usage, the network transmission volume represents the total amount of data that needs to be transmitted between networks during the execution of the scheme, and the execution time estimate is based on historical execution statistics and performance model prediction. The three are combined linearly to obtain the execution cost, and the weights are dynamically adjusted according to the current bottleneck. For example, in a network bandwidth limited environment, the weight of network transmission volume will increase. For an aggregate query, the resource consumption of the parallel number adjustment scheme is 85 (normalized value 0.85), the network transmission volume is 200MB (normalized value 0.6), and the estimated execution time is 150ms (normalized value 0.3). The weighted execution cost is 0.85x0.3+0.6x0.5+0.3x0.2=0.605. By comparing the execution costs of all optimization schemes, the scheme with the minimum execution cost is selected as the optimized query plan. In the above example, if the execution cost of the calculation pushdown scheme is 0.42, which is lower than other schemes, the calculation pushdown scheme is selected to execute the query.

[0132] The optimized query plan is executed to obtain the query result. During execution, detailed performance statistics are recorded, including execution time, resource usage, intermediate result size, etc. These statistics are stored in the distributed query routing table for subsequent query routing decisions and optimization. Performance statistics are stored in JSON format, containing multiple index items such as {"execution_time": 135, "cpu_usage": 65, "memory_usage": 420, "io_operations": 850, "rows_processed": 12500}. The query result is packaged according to the specified format (such as JSON, XML, CSV, etc.) requested by the client, and necessary metadata (such as query time, data version, etc.) is added, and returned to the requester through the original request connection. For pagination queries, the identifier of the next page is also generated to facilitate the client to continue obtaining subsequent data.

[0133] When a node's performance continues to decline or fails, the node is automatically marked as unavailable to avoid new queries being routed to it. At the same time, based on historical query patterns, frequently executed query templates are precompiled and cached to reduce parsing and optimization overhead. For time-sensitive applications, a query timeout mechanism is also supported, which returns the partial results obtained and a timeout prompt when the execution time exceeds the maximum waiting time specified by the client, avoiding long waiting time for the client.

[0134] The application realizes efficient query processing of a distributed database through accurate query request analysis and an intelligent routing distribution mechanism; multi-dimensional routing scoring ensures that a query request is directed to an optimal execution node, and adaptive query plan adjustment can timely respond to abnormal execution conditions; performance prediction based on historical statistics and optimization scheme selection enable continuous optimization of a query path, significantly improve query response speed and throughput capacity, and provide efficient and reliable data access support for data-intensive applications.

[0135] In a second aspect of the embodiment of the application, a data decoupling centralized query system based on MQ message triggering is provided, which comprises:

[0136] A first unit is configured to receive a data change message from a message queue;

[0137] A second unit is configured to construct a field dependency relationship of a data table as a directed acyclic graph, divide nodes into real-time verification nodes and delayed verification nodes based on data integrity requirements, perform data consistency verification on the real-time verification nodes, perform asynchronous consistency verification on the delayed verification nodes, trigger data correction and re-perform consistency verification when detecting that node data is inconsistent, push a transaction identifier carrying a timestamp after consistency verification, the transaction identifier is marked with a unique identifier code to mark the data change message, perform priority identification on concurrent data change operations based on the transaction identifier in combination with a scheduling of a token bucket algorithm, and store the data change operation and the corresponding transaction identifier that pass verification and scheduling to a central database;

[0138] A third unit is configured to collect data access hotspot information of the central database by using a Bloom filter, dynamically adjust data sharding and cache distribution according to the data access hotspot information, and generate a distributed query routing table;

[0139] A fourth unit is configured to analyze and route distribute a query request based on the distributed query routing table, trigger query plan adjustment when a query response time exceeds a preset time threshold, and return data results obtained from the central database according to the adjusted query plan to a requester.

[0140] In a third aspect of the embodiment of the application, an electronic device is provided, which comprises:

[0141] A processor;

[0142] A memory for storing processor-executable instructions;

[0143] The processor is configured to invoke instructions stored in the memory to execute the method described above.

[0144] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method described above.

[0145] The present application can be a method, an apparatus, a system, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present application.

[0146] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for decoupled centralized query based on MQ message trigger, characterized in that, Comprising: receiving data change messages from a message queue; constructing field dependency relationships of a data table as a directed acyclic graph, dividing nodes into real-time verification nodes and delayed verification nodes based on data integrity requirements, verifying data consistency for the real-time verification nodes, verifying asynchronous consistency for the delayed verification nodes, triggering data correction and re-verifying consistency when detecting inconsistent node data, specifically including: constructing field dependency relationships of a data table as a directed acyclic graph, the directed acyclic graph including a field node set and a dependency relationship edge set; calculating node importance scores based on the out-degree, centrality, and business importance of field nodes, dividing field nodes into real-time verification nodes and delayed verification nodes according to the node importance scores; performing local consistency verification and global consistency verification on the real-time verification nodes, the local consistency verification being based on the deviation rate of the current value of a node from the expected value, the global consistency verification being based on the deviation of the current value of a node from the weighted derived value of all in-degree neighbor nodes; performing batch asynchronous verification on the delayed verification nodes according to verification time intervals, dividing delayed verification nodes whose time difference between the last verification time and the current time exceeds a preset time interval into the same batch for verification; when detecting inconsistent target node data, obtaining an upstream node set that directly affects the target node, calculating the correction amount that needs to be adjusted for the upstream nodes and the target node respectively, and determining a correction strategy according to the correction amount; determining an affected node set based on the dependency relationships of the directed acyclic graph, re-verifying the nodes in the affected node set according to dependency topological sorting, and updating node state markers; after consistency verification, pushing transaction identifiers carrying timestamps, the transaction identifiers using unique identification codes to mark the data change messages; scheduling concurrent data change operations based on the transaction identifiers and token bucket algorithms; storing data change operations that pass verification and scheduling and corresponding transaction identifiers to a central database; collecting data access hotspot information of the central database using a Bloom filter, dynamically adjusting data sharding and cache distribution according to the data access hotspot information, and generating a distributed query routing table; analyzing and routing query requests based on the distributed query routing table; when the query response time exceeds a preset time threshold, triggering query plan adjustment, and returning data results obtained from the central database according to the adjusted query plan to the requestor.

2. The method of claim 1, wherein, The step of obtaining an upstream node set that directly affects the target node, calculating the correction amount that needs to be adjusted for the upstream nodes and the target node respectively, and determining a correction strategy according to the correction amount includes: Obtain the historical correction record of the target node, calculate the historical correction frequency and correction amplitude of each upstream node; obtain the service weight of each node in the upstream node set, which is determined based on the service attribute of the node; combine the service weight with the historical correction frequency and correction amplitude to calculate the correction cost of each upstream node, sort the upstream nodes according to the numerical value of the correction cost and select the upstream node with the minimum correction cost for correction; based on the current value of the selected upstream node and the dependency relationship of the target node, combine the mean value of the historical correction amplitude to calculate the correction amount of the upstream node; Obtain the deviation between the real-time data value and the historical stable value when the upstream node correction operation is executed, and obtain the correction amount of the target node through weighted calculation; when the correction amount of the target node exceeds the preset correction amount threshold, split the correction operation into multiple sub-correction steps, and the correction amount of each sub-correction step does not exceed the preset correction amount threshold; Before executing the correction operation, control the correction amplitude based on the number of affected nodes; maintain the correction log during the correction process, and perform compensation operation when an exception is detected.

3. The method of claim 1, wherein, After consistency checking, the transaction identifier carrying the timestamp is pushed, and the step of marking the data change message with a unique identifier code includes: Obtain the current timestamp and the source identifier of the data change message; generate a unique identifier code based on the UUID algorithm, combine the timestamp and the source identifier to obtain a transaction identifier; associate the transaction identifier to the data change message in the form of key-value pair; Build a transaction identifier index table, which includes the transaction identifier as the primary key, the associated message for tracking data source, the generation time for time sequence processing, and the message state for process control; when the data change message completes consistency checking, update the message state to the checked state.

4. The method of claim 1, wherein, The step of identifying the priority of concurrent data change operations based on the transaction identifier and combining the scheduling of the token bucket algorithm includes: Obtain the time-sensitive attribute, spatial locality attribute and data dependency attribute of the target data change operation; calculate the time urgency based on the time difference between the time-sensitive attribute and the deadline, calculate the resource occupancy degree based on the spatial locality attribute and system resource utilization, and calculate the dependency chain complexity based on the data dependency attribute and dependency chain information; combine the time urgency, resource occupancy degree and dependency chain complexity by weighting to obtain the scheduling priority; Build a multi-level token bucket corresponding to the scheduling priority; dynamically adjust the token generation rate based on the system load level; cascade allocate the remaining tokens of the high-priority token bucket according to the request proportion of the low-priority token bucket; select the target data change operation based on the token number of each level token bucket to form a set of operations to be executed; Detecting a deadlock in executing the set of to-be-executed operations, when detecting the deadlock, setting a maximum value of an original priority and a priority accumulated due to blocking as a temporary priority for each data change operation on a deadlock chain; calculating a compensation cost, when the compensation cost is less than a preset cost threshold, re-executing the set of to-be-executed operations according to an execution order adjusted based on the temporary priority.

5. The method of claim 1, wherein, The steps of collecting data access hotspot information of the central database by using a Bloom filter, dynamically adjusting data sharding and cache distribution according to the data access hotspot information, and generating a distributed query routing table include: The steps of collecting data access information of the central database by using a counting Bloom filter, updating a count value of the data access information according to a time interval from a current time, performing logarithmic operation on the updated count value and an average count value of the data access information to obtain a data access hotspot level, and generating the distributed query routing table include: Based on a weighted combination of a sharding storage occupancy ratio, a sharding hotspot quantity ratio and a sharding query quantity ratio, a load balancing degree of the data shard is obtained, and a splitting or merging operation of the shard is triggered according to a size relationship between the load balancing degree and a preset balancing degree threshold. According to the data access hotspot level, cache spaces are allocated among a plurality of cache layers, a size of the cache space is proportional to an exponential value of the data access hotspot level, and a cache item score is obtained by performing weighted calculation on an access quantity, a latest access time and an access overhead of cache data. The data access hotspot level, the load balancing degree and the cache item score are taken as routing features to construct the distributed query routing table, a decision weight of each routing item is calculated, the decision weight is obtained by multiplying a ratio of a hotspot level of a node corresponding to the routing item to hotspot levels of all nodes, a load balancing degree of the node and a cache item score of the node, and a path with the highest decision weight is selected as a query routing path.

6. The method of claim 1, wherein, Based on the distributed query routing table, a query request is parsed and routed and distributed; when a query response time exceeds a preset time threshold, a query plan adjustment is triggered, and data results are obtained from the central database according to the adjusted query plan and returned to a requester. A query request is parsed to obtain a query field set and a query condition set, and routing items matched with the query field set and the query condition set are obtained from the distributed query routing table to form a candidate routing item set. A routing score of each routing item in the candidate routing item set is calculated, the routing score is calculated based on a combination of a data coverage ratio, a load balancing degree and a network transmission delay time of a node corresponding to the routing item, and a node corresponding to a routing item with the highest routing score is selected as a query execution node. A time consumption value in a query execution process is obtained, when the time consumption value exceeds a preset time threshold, candidate optimization scheme sets are generated based on execution performance statistical information, the candidate optimization scheme sets include a parallel quantity adjustment scheme, a node reallocation scheme, a calculation pushdown scheme and an approximate calculation scheme, and an execution cost of each optimization scheme is calculated based on a combination of resource consumption, network transmission quantity and execution time estimation value. Select the optimization scheme with the minimum execution cost as the optimized query plan and execute to obtain a query result; store performance statistical information of the query execution node into the distributed query routing table; and return the query result to the requester.

7. A data decoupled centralized query system based on MQ message triggered for implementing the method of any of the preceding claims 1-6, characterized in that, The method comprises the following steps: A first unit is configured to receive a data change message from a message queue; A second unit is configured to construct a field dependency relationship of a data table as a directed acyclic graph, divide nodes into real-time verification nodes and delayed verification nodes based on data integrity requirements, perform data consistency verification on the real-time verification nodes, perform asynchronous consistency verification on the delayed verification nodes, trigger data correction and re-perform consistency verification when detecting that node data is inconsistent; After consistency verification, a transaction identifier carrying a timestamp is pushed, the transaction identifier is marked with a unique identifier code to mark the data change message; Based on the transaction identifier, a priority identifier is marked for concurrent data change operations in combination with a scheduling of a token bucket algorithm; the data change operations that pass verification and scheduling and corresponding transaction identifiers are stored in a central database; A third unit is configured to collect data access hotspot information of the central database using a Bloom filter, dynamically adjust data sharding and cache distribution according to the data access hotspot information, and generate a distributed query routing table; A fourth unit is configured to parse and route distribute a query request based on the distributed query routing table; when a query response time exceeds a preset time threshold, trigger query plan adjustment, and obtain data results from the central database according to the adjusted query plan and return the data results to the requester.

8. An electronic device, comprising: The method comprises the following steps: A processor; A memory for storing processor-executable instructions; The processor is configured to invoke the instructions stored in the memory to execute the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 6.

Images