Data multi-centralized management method and system based on DOA

Abstract

The application relates to the technical field of distributed data processing and cloud computing, and discloses a data multi-center management method and system based on DOA. The method comprises the following steps: performing semantic decomposition on a query request to obtain a sub-query task set; generating an initial execution plan according to real-time performance data of each data center and the sub-query task set; monitoring an execution process of the initial execution plan, identifying a transmission bottleneck, and dynamically adjusting the execution plan according to the transmission bottleneck; and integrating all sub-query results to generate a final query response. Through the establishment of a dynamic decomposition and adaptive routing planning mechanism for the query, the application solves the problem of low query efficiency caused by rigid management strategies in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed data processing and cloud computing technology, and particularly relates to a data multi-central management method and system based on DOA. BACKGROUND

[0002] At present, in the field of big data processing and data-intensive applications, such as medical image analysis or Internet of Things monitoring system, efficient management of massive cross-regional data has become the core support for promoting industry innovation. DOA, which stands for data-oriented architecture, is a distributed system architecture paradigm that takes data as the core, emphasizes data assetization, service and collaboration. It refers to the ability of data multi-central management and collaborative processing.

[0003] In one prior art, the data management scheme often relies on a unified global view to coordinate resources and process queries. This scheme, when faced with a multi-center environment, its routing decision is easy to deviate from the actual distribution pattern of data, and ignores the dynamic differences in computing power between centers, resulting in rigid query allocation strategies that are difficult to adapt to real-time changing environments. When processing a complex query that requires data interaction across multiple centers, this static management method will amplify network latency and uneven load problems, ultimately making the entire query process inefficient.

[0004] Therefore, there is a problem of low query efficiency in the prior art. SUMMARY

[0005] The present application provides a data multi-central management method and system based on DOA to solve the problem of low query efficiency in the prior art.

[0006] In a first aspect, to solve the above technical problems, the present application provides a data multi-central management method based on DOA, comprising:

[0007] Obtain query request and data object meta-information, and perform semantic decomposition to obtain a set of sub-query tasks;

[0008] According to the set of sub-query tasks, obtain the computing power indicators and dynamic load data of each data center, and perform screening to determine a list of candidate data centers;

[0009] For the list of candidate data centers, comprehensively evaluate network latency and heterogeneous computing power to generate an initial execution plan;

[0010] According to the initial execution plan, execute the set of sub-query tasks and monitor the execution process in real time to identify transmission bottleneck points;

[0011] If the transmission bottleneck point is identified, dynamically adjust the initial execution plan to generate an optimized execution plan;

[0012] Integrating all the subquery results obtained after executing the initial execution plan or the optimized execution plan, and generating a final query response by using an aggregation operation.

[0013] Preferably, the query request and the data object meta-information are obtained, and semantic decomposition is performed to obtain a subquery task set, including:

[0014] The query request and the data object meta-information are obtained from a multi-center environment;

[0015] The semantic structure of the query request is parsed and the internal correlation features are extracted to obtain an internal correlation feature set;

[0016] According to the internal correlation feature set, subquery division is performed to generate the subquery task set.

[0017] Preferably, according to the subquery task set, the computing capability indicators and the dynamic load data of each data center are obtained, and screening is performed to determine a candidate data center list, including:

[0018] The computing capability indicators and the dynamic load data of each data center are obtained;

[0019] According to the computing capability indicators, the dynamic load data, and the subquery task set, the matching degrees of each data center and the subquery task are calculated to obtain a matching degree score;

[0020] According to the matching degree score, the data centers with a matching degree score higher than a preset matching degree threshold value are determined as a preliminary candidate center set;

[0021] The data centers in the preliminary candidate center set are sorted according to the matching degree score to generate the candidate data center list.

[0022] Preferably, for the candidate data center list, network delay and heterogeneous computing capability are comprehensively evaluated to generate an initial execution plan, including:

[0023] The real-time data and the distribution pattern parameters of each data center in the candidate data center list are obtained, and a load balancing indicator is calculated to obtain a load balancing score;

[0024] According to the load balancing score, the data centers with a load balancing score higher than a preset load balancing threshold value are determined as a preferred center set;

[0025] The network delay parameters of each data center in the preferred center set are obtained, and sorting is performed to obtain a lowest delay center set;

[0026] According to the data throughput and computing resource indicators of the minimum delay center set, a routing allocation priority is calculated to obtain a priority ranking list;

[0027] According to the priority ranking list, routing allocation is performed to generate the initial execution plan.

[0028] Preferably, the initial execution plan is executed according to the set of subquery tasks, and the execution process is monitored in real time to identify transmission bottleneck points, including:

[0029] During the execution of the set of subquery tasks, intermediate data transmission records are obtained from the multi-center network;

[0030] The intermediate data transmission records are analyzed to obtain real-time data traffic;

[0031] The real-time data traffic is compared with a preset traffic threshold, and network nodes with real-time data traffic exceeding the preset traffic threshold are determined as the transmission bottleneck points.

[0032] Preferably, if the transmission bottleneck points are identified, the initial execution plan is dynamically adjusted to generate an optimized execution plan, including:

[0033] Real-time variation data of the nodes corresponding to the transmission bottleneck points are obtained, and node loads are analyzed from the real-time variation data;

[0034] If the node load exceeds a preset path switching threshold, a set of backup paths is determined;

[0035] According to the set of backup paths, routing optimization parameters are recalculated;

[0036] According to the routing optimization parameters, the optimized execution plan is generated.

[0037] Preferably, all subquery results obtained after executing the initial execution plan or the optimized execution plan are integrated, and an aggregation operation is performed to generate a final query response, including:

[0038] All the subquery results are obtained from the multi-center network nodes to form a set of subquery results;

[0039] The semantic integrity of each subquery result in the set of subquery results is determined, and if the semantic integrity is lower than a preset integrity threshold, semantic completion is performed;

[0040] According to a preset association rule, data objects in the set of subquery results after semantic completion are integrated to obtain a set of aggregated data objects;

[0041] According to the set of aggregated data objects, a final query response is generated.

[0042] In a second aspect, the present application provides a DOA-based data multi-center management device, comprising:

[0043] A query decomposition module is configured to obtain a query request and data object meta-information, and perform semantic decomposition to obtain a set of sub-query tasks.

[0044] A candidate screening module is configured to obtain computing capability indicators and dynamic load data of each data center according to the set of sub-query tasks, and perform screening to determine a candidate data center list.

[0045] A plan generation module is configured to comprehensively evaluate network delay and heterogeneous computing capability for the candidate data center list, and generate an initial execution plan.

[0046] An execution monitoring module is configured to execute the set of sub-query tasks according to the initial execution plan, and monitor the execution process in real time to identify a transmission bottleneck point.

[0047] A dynamic adjustment module is configured to dynamically adjust the initial execution plan to generate an optimized execution plan if the transmission bottleneck point is identified.

[0048] A result aggregation module is configured to integrate all sub-query results obtained after execution of the initial execution plan or the optimized execution plan, and perform aggregation operation processing to generate a final query response.

[0049] In a third aspect, the present application further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the DOA-based data multi-center management method according to any one of the above.

[0050] In a fourth aspect, the present application further provides a computer readable storage medium, comprising a stored computer program, wherein the computer program controls a device where the computer readable storage medium is located to execute the DOA-based data multi-center management method according to any one of the above when the computer program is running.

[0051] Compared with the prior art, the present application has the following beneficial effects:

[0052] (1) This invention decomposes query requests semantically and dynamically generates execution plans by combining the real-time computing power, network latency and load status of each node in a multi-center network. This enables the allocation and routing decisions of queries to closely match the actual distribution pattern of data and the real-time performance of nodes. It solves the problems of rigid routing and resource allocation that are out of touch with the actual situation caused by relying on static global views in the prior art, and significantly improves the initial execution efficiency of complex queries in a distributed environment.

[0053] (2) This invention monitors data transmission traffic in real time during execution and identifies and locates transmission bottlenecks based on preset thresholds. Once a bottleneck is detected, a rerouting mechanism can be triggered immediately, and the execution path can be dynamically adjusted according to the latest network status. This forms a closed-loop control system from performance monitoring to fault self-healing, which solves the problems of slow response and lack of elastic recovery capability in the face of network fluctuations and sudden changes in node load in the prior art, and ensures the stability and high availability of data processing tasks.

[0054] (3) When integrating the subquery results distributed across nodes, this invention introduces a semantic integrity verification and completion mechanism, which can proactively identify and repair information loss or context inconsistency problems that may occur due to distributed execution before data aggregation, ensuring the internal logical unity of the final aggregated data. This solves the problem that the existing technology only performs simple data splicing when merging distributed results, which is prone to producing errors or incomplete results, and fundamentally improves the accuracy and reliability of the final query response. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the data multi-center management method based on DOA provided in the first embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of a DOA-based multi-center data management system provided in the second embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Reference Figure 1 The first embodiment of the present invention provides a data multi-center management method based on DOA, including the following steps:

[0059] S11: Obtain the query request and data object metadata, and perform semantic decomposition to obtain a set of subquery tasks;

[0060] S12, Based on the set of sub-query tasks, obtain the computing power indicators and dynamic load data of each data center, and filter them to determine the candidate data center list;

[0061] S13, For the candidate data center list, comprehensively evaluate network latency and heterogeneous computing capabilities, and generate an initial execution plan;

[0062] S14, Execute the set of sub-query tasks according to the initial execution plan, and monitor the execution process in real time to identify transmission bottlenecks;

[0063] S15, if the transmission bottleneck is identified, the initial execution plan is dynamically adjusted to generate an optimized execution plan;

[0064] S16, integrate all subquery results obtained after executing the initial execution plan or the optimized execution plan, and process them using aggregation operations to generate the final query response.

[0065] In step S11, the query request and data object metadata are obtained and semantically decomposed to obtain a set of subquery tasks, including:

[0066] Obtain the query request and the data object metadata from a multi-center environment;

[0067] The semantic structure of the query request is parsed and the inherent correlation features are extracted to obtain the inherent correlation feature set;

[0068] Based on the intrinsic association feature set, subqueries are divided to generate the subquery task set.

[0069] In one implementation, the query request issued by the user and the data object metadata describing the data distribution status of the entire distributed network are first obtained from the multi-center environment.

[0070] It should be noted that the query request supports both natural language text and Structured Query Language (SQL) statements. The data object metadata is a structured dataset, configured by the database administrator during initialization according to predetermined data sharding and governance strategies, and stored in a centralized metadata repository for the system to retrieve at runtime. The core component of this dataset is a routing rule base, and it may also include attributes such as the storage location of data tables and the data holding scope of each data center.

[0071] For example, a user's natural language query request might be: "Retrieve all records of orders with a total amount greater than 1000 yuan in January 2025". Meanwhile, the metadata of the data objects retrieved from the metadata repository might contain the following rules: "The order table is distributed across two data centers in Shanghai and Beijing; the Shanghai data center stores data from 2024 and earlier; the Beijing data center stores data from 2025 and later."

[0072] It is worth noting that after acquiring the raw data, based on the query request, a built-in query parsing engine parses its semantic structure and extracts its inherent correlation features to obtain a structured set of inherent correlation features. This parsing process transforms different forms of input queries into a unified and standardized intermediate representation, namely the set of inherent correlation features, enabling subsequent modules to process them uniformly.

[0073] In one implementation, if the query request is an SQL statement, the query parsing engine can be a standard SQL syntax parser. This parser generates an abstract syntax tree (AST) through syntax analysis, then traverses the AST, extracts key query parameters, and formats them into the intrinsic relational feature set. If the query request is natural language, the query parsing engine can be a pre-trained model based on natural language processing techniques.

[0074] It should be noted that the model construction and training process based on natural language processing technology is as follows:

[0075] First, a training dataset is constructed. This dataset is created by acquiring tens of thousands of historical query logs, and data engineers manually annotate each natural language query to produce data pairs consisting of a structured set of inherently related features.

[0076] Subsequently, the data pairs are input into a pre-defined deep learning-based sequence labeling and relation extraction model structure for training. It should be noted that the model structure may specifically include sequentially connected embedding layers, bidirectional long short-term memory (Bi-LSTM) layers, and conditional random field (CRF) output layers.

[0077] The training process uses the cross-entropy loss function to calculate the error between the model's predictions and the ground truth labels, and iteratively adjusts the network weights using the backpropagation algorithm driven by the Adam optimizer. Training terminates when the model's loss value on the independent validation set falls below a preset convergence threshold. It should be noted that this convergence threshold is determined through multiple training experiments, recording the model's validation set loss value at different training stages and its query parsing accuracy on the independent test set; a loss value that achieves a preset business requirement (e.g., 95%) on the test set while maintaining training efficiency is selected as the preset convergence threshold (e.g., 0.01).

[0078] For example, after parsing the aforementioned natural language query request, the following set of inherent relational features can be obtained: {data table: "order table", filter condition: [{field: "time", operator: "starting from", value: "2025-01"}, {field: "amount", operator: "greater than", value: "1000"}]}.

[0079] In another implementation, based on the intrinsic association feature set generated in the preprocessing and combined with the data object metadata, a semantic decomposition module is used to partition the subqueries, ultimately generating the subquery task set. It should be noted that this partitioning process is performed based on a preset routing rule base within the data object metadata.

[0080] The routing rule base is predefined by the database administrator based on business needs and data physical distribution strategies. Each rule can be a structured entry containing four fields: "table name", "shard key", "shard condition", and "target data center". For example, {table name: "order table", shard key: "time", shard condition: "year < 2025", target data center: "Shanghai"}.

[0081] The semantic decomposition module's matching process involves traversing each filter condition in the intrinsic association feature set. If a field of a filter condition matches the "sharding key" of a rule, the module substitutes the value from the filter condition into the "sharding condition" expression of that rule for logical evaluation. If the evaluation result is true, it determines that the query operation related to that filter condition should be routed to the "target data center" specified by that rule.

[0082] It should be noted that after determining the target data center, this embodiment also includes a query statement generation step. This step transforms the intrinsic association feature set into a specific query statement that can be executed in the target data center. In a specific implementation, this generation step is completed by a query statement generator, whose processing includes: extracting data table names from the intrinsic association feature set and generating FROM clauses of Structured Query Language; traversing all filtering conditions in the intrinsic association feature set and mapping the fields, operators, and values ​​of each condition to a standard condition expression in the WHERE clause; and combining all generated clauses to form a complete, executable query statement.

[0083] For example, the time condition "2025-01" in the aforementioned intrinsic association feature set is matched with the rule {table name: "order table", shard key: "time", shard condition: "year>=2025", target data center: "Beijing"} in the data object metadata. The year "2025" is extracted from the time value "2025-01" and substituted into the shard condition "year>=2025" for calculation, and the result is true. Therefore, it is determined that the query is only related to the Beijing data center. Subsequently, the query statement generator generates the query statement "SELECT * FROM order table WHERE time>='2025-01-01' AND time<'2025-02-01' AND amount>1000" based on this intrinsic association feature set, and encapsulates it together with the target data center "Beijing" to form a complete subquery task.

[0084] In step S12, based on the sub-query task set, the computing power indicators and dynamic load data of each data center are obtained, filtered, and a candidate data center list is determined, including:

[0085] Obtain the computing power metrics and dynamic load data of each data center;

[0086] Based on the computing power index, the dynamic load data, and the set of subquery tasks, the matching degree between each data center and the subquery task is calculated to obtain a matching degree score.

[0087] Based on the matching score, data centers with matching scores higher than a preset matching threshold are identified as a preliminary candidate data center set.

[0088] The data centers in the preliminary candidate data center set are sorted according to the matching score to generate the candidate data center list.

[0089] In one implementation, monitoring modules deployed on various nodes in a multi-center network are used to collect and acquire the computing power indicators and dynamic load data of each data center in real time.

[0090] It should be noted that the computing power metrics are a set of parameters used to quantify the static processing capacity of a data center, which may include node CPU utilization, available memory, and disk I / O rate. The dynamic load data, on the other hand, is a set of parameters reflecting the current real-time workload of the data center, which may include the number of currently executing query tasks and network bandwidth utilization.

[0091] It is worth noting that after obtaining the real-time performance data of each data center, the matching degree between each data center and the subquery task is calculated to obtain a quantified matching degree score. In a specific implementation, this matching degree score is calculated using a preset weighted scoring model. This model integrates three dimensions of indicators: data locality, CPU availability, and memory availability. Its calculation formula can be expressed as:

[0092]

[0093] in, The final match score is given. This is the data locality score. If the data required by the subquery is stored in the current data center, the value is 1; otherwise, it is 0. The CPU availability score can be calculated as (1 minus the current CPU utilization). The memory availability score can be calculated as (current available memory divided by total memory). , and Each indicator has a preset weighting coefficient, and their sum is 1. It should be noted that these weighting coefficients are determined based on multiple linear regression analysis of historical query task performance data. The determination process includes: collecting a dataset containing thousands of historical query samples, each sample containing data from the task execution... , , The metric value and the final actual query time; with query time as the dependent variable, , , Using the variables as independent variables, establish a linear regression model to obtain the regression coefficients of each independent variable's influence on query time. Normalize the absolute values ​​of each regression coefficient, calculate the sum of all absolute values, and then divide the absolute value of each coefficient by this sum to ensure that the sum of all weight coefficients equals 1. This yields the final weight coefficients (e.g., ...). =0.5, =0.3, =0.2).

[0094] For example, suppose a subquery task is "retrieve order data for January 2025". The Beijing data center stores data from 2025 (…). =1), CPU utilization is 50% ( =0.5), 8GB of memory available / 16GB total ( =0.5), its matching score is calculated to be 0.75. However, the Shanghai data center does not have this data ( If the score is 0, the score will decrease significantly.

[0095] In another implementation, based on the calculated matching score, data centers with scores higher than a preset matching threshold are identified as the initial candidate center set. It should be noted that the preset matching threshold is determined as follows: Through statistical analysis of a large amount of scheduling decision data during the system's historical normal operation, the matching score distribution of all successfully scheduled and efficiently completed tasks is recorded; the value corresponding to the 80th percentile of this distribution is selected as the preset matching threshold (e.g., 0.8). This method ensures that only high-quality nodes that have historically performed in the top 20% are initially selected; however, this threshold is a dynamically configurable strategy parameter, not static, to avoid unreasonable resource utilization; for example, the median (50%) or average of the historical score distribution can also be selected.

[0096] Finally, in this embodiment, the data centers in the preliminary candidate center set are sorted in descending order according to their respective matching scores using an efficient sorting algorithm (such as quicksort algorithm), ultimately generating an ordered list of candidate data centers with clear priorities for subsequent steps to perform more refined routing planning.

[0097] In step S13, for the candidate data center list, network latency and heterogeneous computing capabilities are comprehensively evaluated to generate an initial execution plan, including:

[0098] Obtain real-time data and distribution pattern parameters of each data center in the candidate data center list, and calculate the load balancing index to obtain a load balancing score.

[0099] Based on the load balancing score, data centers with load balancing scores higher than the preset load balancing threshold are identified as the preferred data center set.

[0100] Obtain the network latency parameters of each data center in the preferred set of centers, sort them, and obtain the set of centers with the lowest latency;

[0101] Based on the data throughput and computing resource metrics of the lowest latency center set, the routing allocation priority is calculated to obtain a priority ranking list.

[0102] The route is assigned according to the priority sorting list, and the initial execution plan is generated.

[0103] In one implementation, this embodiment initiates a refined multi-stage evaluation process for the candidate data center list generated in the preprocessing. First, real-time data and distribution pattern parameters of each data center in the candidate data center list are obtained, and load balancing metrics are calculated to obtain a quantified load balancing score.

[0104] It should be noted that the real-time data may include the current task queue length, CPU utilization, and memory usage of each data center. The distribution pattern parameters may include the location information of data shards. In a specific implementation, the load balancing score is calculated as a normalized idle rate index, and its calculation formula is:

[0105]

[0106] in, Score the load balancer; , and These are the normalized task queue length, CPU utilization, and memory usage, respectively, obtained by dividing the real-time value by a preset maximum allowable value. It should be noted that the maximum allowable value is determined as follows: for CPU utilization and memory usage, the maximum allowable value is determined based on the node's physical hardware specifications (100% and total memory, respectively); for the task queue length, the maximum allowable value is determined by conducting offline stress tests on the system to find the queue length corresponding to the point where the system response time begins to show a significant inflection point, and using this as a baseline value. , and These are the corresponding weighting coefficients, and their sum is 1. They are pre-configured by the system administrator based on the impact of different load metrics on system performance. It should be noted that the configuration process includes: conducting performance profiling experiments on typical business scenarios to analyze the correlation between the growth of various load metrics (task queue, CPU, memory) and key system performance metrics (such as query response latency); and assigning higher weighting coefficients to the load metrics that have the most significant impact on performance based on the experimental results.

[0107] In another implementation, the calculated load balancing score is compared with a preset load balancing threshold, and data centers with scores higher than the threshold are identified as the preferred data center set. It should be noted that the preset load balancing threshold is determined as follows: by analyzing historical system performance data, the load balancing score distribution of all participating nodes is statistically analyzed within multiple time windows (e.g., 5 minutes per window) where the overall system response time is optimal; the value corresponding to the 75th percentile of this distribution is selected as the preset load balancing threshold.

[0108] It is worth noting that after obtaining the preferred set of data centers, the network latency parameters of each data center in the set are further obtained and sorted in ascending order of network latency to obtain the lowest latency set of data centers. Subsequently, for the nodes in the lowest latency set of data centers, their data throughput and computing resource indicators are extracted, and a weighted average algorithm is used to calculate the final routing allocation priority, resulting in the priority ranking list. In a specific implementation, the priority calculation formula can be expressed as:

[0109]

[0110] in, Assign priority to routes; This is a normalized data throughput metric. This is a normalized computing resource indicator (obtained by normalizing and weighting sub-items such as CPU computing power and memory capacity). It should be noted that the normalization process uses a max-min normalization method, scaling the indicator values ​​of all nodes to a uniform range of [0,1]. and These are the corresponding preset weights, and their sum is 1. The weights are set according to the typical needs of the business scenario. For example, for data-intensive queries, a weight can be set to... =0.7, =0.3, prioritize nodes with strong data transmission capabilities.

[0111] Finally, this embodiment uses a greedy scheduling algorithm to perform routing allocation based on the priority sorting list, generating the initial execution plan. This algorithm traverses the set of subquery tasks, and for each subquery, selects the data center with the highest current priority and not yet saturated resources from the priority sorting list for allocation. The final generated initial execution plan is a data structure containing a clear mapping relationship between each subquery task and the allocated target data center.

[0112] For example, suppose the candidate data center list includes three centers: Beijing, Shanghai, and Guangzhou. In the first stage, the load balancing scores for the three centers are calculated to be 0.9, 0.6, and 0.85, respectively. If the preset load balancing threshold is 0.8, then Beijing and Guangzhou are selected as the preferred centers. In the second stage, the network latency for Beijing and Guangzhou is found to be 10ms and 15ms, respectively. After sorting, the set of centers with the lowest latency is obtained. In the third stage, assuming that Beijing's data throughput and computing resource indicators are better than Guangzhou's, its routing priority is calculated to be 0.92, higher than Guangzhou's 0.88, forming a priority ranking list. Finally, the greedy algorithm will assign the highest priority subquery task to the Beijing data center, generating an initial execution plan.

[0113] In step S14, the set of sub-query tasks is executed according to the initial execution plan, and the execution process is monitored in real time to identify transmission bottlenecks, including:

[0114] During the execution of the set of subquery tasks, intermediate data transmission records are obtained from the multi-center network;

[0115] The real-time data flow is obtained by analyzing the intermediate data transmission records.

[0116] The real-time data traffic is compared with a preset traffic threshold, and network nodes whose real-time data traffic exceeds the preset traffic threshold are identified as transmission bottlenecks.

[0117] In one implementation, during the execution of the sub-query task set according to the initial execution plan, the runtime logs of each node are scanned in real time, and preset log parsing rules are applied to match and extract entries related to data transmission from the runtime logs, forming the intermediate data transmission record. It should be noted that the preset log parsing rules can be a series of regular expressions or text matching patterns, pre-configured according to the output format of the runtime logs.

[0118] It should be noted that the intermediate data transmission record is a structured dataset, in which each record may contain fields such as the source node address, destination node address, data packet size, transmission timestamp, and transmission time.

[0119] It is worth noting that a traffic monitoring module continuously analyzes the intermediate data transmission records to calculate the real-time data traffic of each node. In a specific implementation, the real-time data traffic is calculated as follows: within a preset sliding time window (e.g., 1 second), the size of all data packets flowing through a certain network node's intermediate data transmission records is summed, and then divided by the length of the time window to obtain the real-time data traffic value of that node in megabytes per second (MB / s).

[0120] In another implementation, the calculated real-time data traffic is compared with a preset traffic threshold, and any network node whose real-time data traffic exceeds this threshold is identified as the transmission bottleneck. It should be noted that the preset traffic threshold is determined as follows: A probability distribution map of the traffic indicators is plotted by performing long-term statistical analysis on network traffic data during the system's historical normal operation (i.e., periods without user-reported delays or timeout errors); the traffic value corresponding to the 95th percentile of this distribution map is selected as the preset traffic threshold. This threshold represents a high-water mark for the system's healthy state; exceeding this threshold is considered a statistically abnormal event, indicating a high probability that the node has become or is about to become a performance bottleneck.

[0121] For example, suppose that based on historical data analysis, a preset traffic threshold of 1000 MB / s is determined. During a query execution, log analysis reveals that the Shanghai node transmitted 1200 MB of data in the most recent second, calculating its real-time data traffic to be 1200 MB / s. Since 1200 MB / s exceeds the 1000 MB / s threshold, the Shanghai node is marked as the transmission bottleneck, triggering a subsequent dynamic adjustment process.

[0122] In step S15, if the transmission bottleneck is identified, the initial execution plan is dynamically adjusted to generate an optimized execution plan, including:

[0123] Obtain real-time variation data of the node corresponding to the transmission bottleneck point, and analyze the node load from the real-time variation data;

[0124] If the node load exceeds a preset path switching threshold, then a set of backup paths is determined.

[0125] Based on the set of alternative paths, the routing optimization parameters are recalculated;

[0126] Based on the routing optimization parameters, the optimized execution plan is generated.

[0127] In one implementation, if the transmission bottleneck is identified during the preceding processing, a dynamic adjustment process is immediately initiated. This process first targets the network node corresponding to the transmission bottleneck and obtains its latest real-time variation data through its monitoring module.

[0128] It should be noted that the real-time variable data refers to a collection of information such as dynamically changing traffic, latency, or node load during network operation. From this data set, a quantified node load value is analyzed and calculated. In a specific implementation, the node load value is calculated using the same weighted model and performance metrics described in S13 for calculating the load balancing score; that is, a weighted sum of CPU utilization, memory usage, and task queue length. This calculation result serves as a single index ranging from 0% to 100%, used to quantify the congestion level of a node.

[0129] It is worth noting that the calculated node load value is compared with a preset path switching threshold. If the node load exceeds the threshold, it indicates that the congestion of that node has seriously affected system performance, and path switching must be performed. At this time, the current multi-center network topology and the real-time load status of all nodes are queried to identify one or more nodes that can perform the same task and whose current load is below the path switching threshold, forming the backup path set.

[0130] It should be noted that the method for determining the preset path switching threshold is as follows: By conducting a series of offline stress tests on the system with gradually increasing load, a curve showing the relationship between key system performance indicators (such as average query response latency) and node load is plotted; the load value corresponding to the "inflection point" on the curve where performance begins to drop sharply is identified, and this load value is set as the preset path switching threshold (e.g., 70%); the threshold is only an example and can be configured and adjusted according to the performance characteristics and business requirements of different systems.

[0131] In another implementation, after the set of backup paths is determined, the routing optimization parameters are recalculated based on the node information in the set. This process may include: for each node in the set of backup paths, re-executing the routing allocation priority calculation process in S13 to obtain a new priority order for the backup paths.

[0132] Finally, this embodiment generates the optimized execution plan based on the newly calculated routing optimization parameters. In a specific implementation, this generation process includes: allowing subquery tasks currently executing at the transmission bottleneck to continue completing, ensuring data processing consistency; extracting all subsequent subquery tasks that have not yet started execution and were originally assigned to the transmission bottleneck from the initial execution plan, and reassigning these tasks to the highest-priority backup node according to the priority of the backup path. This reassigned, new execution plan is the optimized execution plan.

[0133] For example, suppose the Hangzhou node is identified as a transmission bottleneck, and its current node load is calculated to be 85%. This value exceeds the preset path switching threshold of 70%. The Suzhou node, with a current load of only 60%, is then selected as the backup path. Subsequently, the routing optimization parameters are recalculated, and the two sub-query tasks originally allocated to the Hangzhou node are reassigned to the Suzhou node, thereby generating an optimized execution plan that bypasses the congestion at the Hangzhou node. This plan is then deployed and executed.

[0134] In step S16, all subquery results obtained after executing the initial execution plan or the optimized execution plan are integrated and processed using an aggregation operation to generate the final query response, including:

[0135] All subquery results are obtained from the multi-center network nodes to form a subquery result set;

[0136] Determine the semantic completeness of each subquery result in the subquery result set. If the semantic completeness is lower than a preset completeness threshold, perform semantic completion.

[0137] According to the preset association rules, the data objects in the semantically completed subquery result set are integrated to obtain an aggregated data object set;

[0138] The final query response is generated based on the aggregated data object set.

[0139] In one implementation, after all subquery tasks have been executed in each data center, all returned subquery results are obtained from each network node that executed the tasks, and these results are aggregated to form the subquery result set.

[0140] It should be noted that before aggregating the results, a semantic integrity check is first performed on each subquery result in the subquery result set. In one specific implementation, this check is based on a preset data model. This data model is predefined by the system administrator according to business requirements, specifying the necessary association fields and their data types for each type of data object during association operations. The check operation involves iterating through each data record in the subquery result and checking whether the necessary association fields defined in the data model exist and whether their values ​​are not empty. If the proportion of records lacking necessary association fields in a subquery result exceeds a preset integrity threshold, a semantic completion process is initiated for that subquery result.

[0141] It is worth noting that the determination of the preset integrity threshold is based on the specific requirements of the business for data quality. For example, for scenarios requiring precise user behavior analysis, a relatively strict threshold, such as 5%, can be set, allowing a maximum of 5% of records to have missing information. The semantic completion is implemented by, when a record is found to be missing a necessary field (e.g., user ID), using other auxiliary fields present in the record (e.g., session ID or device ID), initiating an internal query to a preset master data table or log database that stores complete related information, retrieving and filling in the missing field information. It should be noted that the master data table or log database is a centralized storage designated as the authoritative data source during the system design phase, such as an enterprise-level customer data platform (CDP) or a unified user session log center, whose data integrity and accuracy are guaranteed by an independent ETL (extract, transform, load) process.

[0142] It should be noted that, to ensure the robustness and high availability of the system, the internal query is set with a preset query timeout threshold. If the query fails or times out, the system can perform a preset number of retries. After all retries fail, a rollback strategy will be initiated, for example, marking the data record as "completeness failed" and removing it from subsequent aggregation processing, while simultaneously recording a system alarm to ensure the data quality of the final output. It should be noted that the query timeout threshold is determined based on statistical analysis of the historical query response time of the master data table; for example, it can be set to 120% of the 99th percentile response time of historical queries, to tolerate network fluctuations while avoiding excessively long wait times. The number of retries is set according to the trade-off between the business's requirements for data consistency and system load; for example, it can be preset to 2 times to handle momentary network failures. Those skilled in the art should understand that the thresholds described in this invention are not fixed and can be configured and adjusted according to the actual system's performance characteristics and business requirements.

[0143] In another implementation, based on preset association rules, all data objects in the verified and completed semantically complete subquery result set are integrated to obtain the aggregated data object set. It should be noted that the association rules are determined based on the intrinsic association feature set generated in S11, which defines the association keys used for join operations between different data objects.

[0144] Finally, in this embodiment, based on the aggregated data object set, a result fusion module generates the final query response according to a preset response format (e.g., JSON or XML). This response not only includes the complete aggregated data but may also selectively include execution summary information for this query, such as total execution time and a list of involved data centers.

[0145] For example, suppose user click data is obtained from the Shanghai node and user browsing duration data is obtained from the Guangzhou node. During semantic integrity verification, it is found that some browsing duration data returned by the Guangzhou node is missing the user ID, but contains the session ID. The semantic completion module then initiates, using the session ID to query the user session log database, finds the corresponding user ID, and adds it to the data. Subsequently, based on the association rule of "consistent user ID," the click data from the Shanghai node is merged with the completed browsing duration data from the Guangzhou node to form a complete user behavior record. Finally, the result fusion module formats this record into a JSON object and returns it to the user as the final query response.

[0146] To facilitate understanding of the present invention, some preferred embodiments of the present invention will be described in further detail below.

[0147] In one implementation, the method of the present invention can be applied to a complete, closed-loop cross-regional order query and analysis scenario. A complete workflow example is as follows:

[0148] Step one: The system receives a natural language query from the user: "Query details of all large orders exceeding 1000 yuan in North China in January 2025." Using a pre-trained natural language processing model, the query is parsed into a structured set of inherent relational features. Subsequently, the semantic decomposition module generates a subquery task targeting the Beijing data center, based on the rule in the metadata that "data from 2025 is stored in the Beijing data center."

[0149] Step two involves acquiring real-time performance metrics for the three data centers in Beijing, Shanghai, and Guangzhou. Since the data for this task is only stored in Beijing, the matching score for the Beijing data center is calculated to be 0.9, significantly higher than the 0.2 scores for Shanghai and Guangzhou. After threshold filtering and sorting, the final candidate data center list prioritizes the Beijing center.

[0150] Step 3: A detailed evaluation was conducted on the Beijing data center, revealing a current load balancing score of 0.92, a network latency of 15ms, and strong data throughput. Based on these metrics, an initial execution plan was generated that explicitly assigned the aforementioned subquery tasks to the Beijing data center for execution.

[0151] Step four: Issue and execute the plan. During execution, the monitoring module detected that the data traffic from the Beijing data center reached 1200MB / s, exceeding the preset traffic threshold of 1000MB / s, and subsequently marked the Beijing node as the transmission bottleneck.

[0152] Step 5: Having identified the bottleneck, the latest load of the Beijing node was immediately retrieved, revealing that its overall load had reached 85%, exceeding the 70% path switching threshold. Further investigation showed that the Tianjin data center, serving as Beijing's disaster recovery center, currently had only 40% load and could be used as a backup path. The routing parameters were then recalculated, generating an optimized execution plan that included instructions to offload some intermediate processing tasks (such as data formatting) to the Tianjin node.

[0153] Step Six: After the nodes in Beijing and Tianjin collaboratively complete the task, collect all subquery results. During verification, it was discovered that some order records were missing the required associated field, "Customer Manager ID." The semantic completion module was activated, using the "Customer ID" from the order to query the Customer Relationship Management (CRM) master data table and complete the missing "Customer Manager ID." Finally, all the completed order records were integrated, formatted into a JSON response, and returned to the user.

[0154] In another implementation, the system deployment architecture of the method of this invention can be configured differently. For example, in the financial or government sectors where security and performance requirements are extremely high, the multiple data centers can be connected via a high-speed, stable dedicated network that does not traverse the public internet. All computing and scheduling are performed in a private cloud environment to ensure the highest data security and lowest network latency. Furthermore, when applied to global internet services, the data centers can be deployed in different availability zones of different public cloud providers, connected via dedicated interconnection channels provided by the cloud providers or the public internet. The method of this invention can adapt to these two vastly different network environments, dynamically adjusting its routing and scheduling strategies to ensure efficient data management under different architectures.

[0155] This invention discloses a multi-center data management method based on Data of Authority (DOA), comprising: acquiring query requests and data object metadata, and performing semantic decomposition to obtain a set of sub-query tasks; acquiring computing power indicators and dynamic load data of each data center based on the set of sub-query tasks, and filtering them to determine a candidate data center list; comprehensively evaluating network latency and heterogeneous computing capabilities for the candidate data center list to generate an initial execution plan; executing the set of sub-query tasks according to the initial execution plan, and monitoring the execution process in real time to identify transmission bottlenecks; if the transmission bottleneck is identified, dynamically adjusting the initial execution plan to generate an optimized execution plan; integrating all sub-query results obtained after executing the initial execution plan or the optimized execution plan, and processing them using aggregation operations to generate a final query response. This invention solves the problem of low query efficiency caused by static management strategies and lack of adaptability to dynamic changes in the distributed environment in existing technologies by dynamically decomposing query requests and performing adaptive routing planning and bottleneck adjustment based on real-time perception of data distribution, node computing power, and network status in a multi-center network, and finally combining semantic integrity verification to perform high-quality aggregation of results.

[0156] Reference Figure 2 The second embodiment of the present invention provides a data multi-center management system based on DOA, including:

[0157] The query decomposition module is used to obtain query requests and data object metadata, and perform semantic decomposition to obtain a set of subquery tasks;

[0158] The candidate filtering module is used to obtain the computing power indicators and dynamic load data of each data center based on the set of subquery tasks, and to filter and determine the candidate data center list.

[0159] The plan generation module is used to generate an initial execution plan based on the candidate data center list by comprehensively evaluating network latency and heterogeneous computing capabilities.

[0160] The execution monitoring module is used to execute the set of sub-query tasks according to the initial execution plan, and to monitor the execution process in real time and identify transmission bottlenecks.

[0161] The dynamic adjustment module is used to dynamically adjust the initial execution plan and generate an optimized execution plan if the transmission bottleneck is identified.

[0162] The result aggregation module is used to integrate all subquery results obtained after executing the initial execution plan or the optimized execution plan, and process them using aggregation operations to generate the final query response.

[0163] It should be noted that the DOA-based data decentralization management system provided in this embodiment of the invention is used to execute all process steps of the DOA-based data decentralization management method described above. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0164] This invention also provides an electronic device. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a DOA-based data decentralization management program. When the processor executes the computer program, it implements the steps described in the various DOA-based data decentralization management method embodiments above, for example... Figure 1 The step S11 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, such as the query decomposition module.

[0165] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0166] The electronic device may be a desktop computer, laptop, handheld computer, or smart tablet, etc. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0167] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting all parts of the electronic device via various interfaces and lines.

[0168] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0169] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0170] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0171] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A data multi-center management method based on DOA, characterized in that, include: Obtain the query request and data object metadata, and perform semantic decomposition to obtain a set of subquery tasks; Based on the set of subquery tasks, obtain the computing power indicators and dynamic load data of each data center, and filter them to determine the candidate data center list; For the candidate data center list, network latency and heterogeneous computing capabilities are comprehensively evaluated to generate an initial execution plan; The set of sub-query tasks is executed according to the initial execution plan, and the execution process is monitored in real time to identify transmission bottlenecks. If the transmission bottleneck is identified, the initial execution plan is dynamically adjusted to generate an optimized execution plan; Integrate all subquery results obtained after executing the initial execution plan or the optimized execution plan, and process them using an aggregation operation to generate the final query response; Specifically, the step of generating an initial execution plan by comprehensively evaluating network latency and heterogeneous computing capabilities for the candidate data center list includes: Obtain real-time data and distribution pattern parameters of each data center in the candidate data center list, and calculate the load balancing index to obtain a load balancing score. Based on the load balancing score, data centers with load balancing scores higher than the preset load balancing threshold are identified as the preferred data center set. Obtain the network latency parameters of each data center in the preferred set of centers, sort them, and obtain the set of centers with the lowest latency; Based on the data throughput and computing resource metrics of the lowest latency center set, the routing allocation priority is calculated to obtain a priority ranking list. The route is assigned according to the priority sorting list, and the initial execution plan is generated. The step of executing the set of sub-query tasks according to the initial execution plan, and monitoring the execution process in real time to identify transmission bottlenecks includes: During the execution of the set of subquery tasks, intermediate data transmission records are obtained from the multi-center network; The real-time data flow is obtained by analyzing the intermediate data transmission records. The real-time data traffic is compared with a preset traffic threshold, and network nodes whose real-time data traffic exceeds the preset traffic threshold are identified as the transmission bottlenecks. The step of dynamically adjusting the initial execution plan and generating an optimized execution plan if the transmission bottleneck is identified includes: Obtain real-time variation data of the node corresponding to the transmission bottleneck point, and analyze the node load from the real-time variation data; If the node load exceeds a preset path switching threshold, then a set of backup paths is determined. Based on the set of alternative paths, the routing optimization parameters are recalculated; Based on the routing optimization parameters, the optimized execution plan is generated.

2. The data multi-center management method based on DOA according to claim 1, characterized in that, The process of obtaining query requests and data object metadata, and performing semantic decomposition to obtain a set of sub-query tasks includes: Obtain the query request and the data object metadata from a multi-center environment; The semantic structure of the query request is parsed and the inherent correlation features are extracted to obtain the inherent correlation feature set; Based on the intrinsic association feature set, subqueries are divided to generate the subquery task set.

3. The data multi-center management method based on DOA according to claim 1, characterized in that, The step involves obtaining the computing power metrics and dynamic load data of each data center based on the sub-query task set, filtering them, and determining a candidate data center list, including: Obtain the computing power metrics and dynamic load data of each data center; Based on the computing power index, the dynamic load data, and the set of subquery tasks, the matching degree between each data center and the subquery task is calculated to obtain a matching degree score. Based on the matching score, data centers with matching scores higher than a preset matching threshold are identified as a preliminary candidate data center set. The data centers in the preliminary candidate data center set are sorted according to the matching score to generate the candidate data center list.

4. The data multi-center management method based on DOA according to claim 1, characterized in that, The process integrates all subquery results obtained after executing the initial execution plan or the optimized execution plan, and processes them using aggregation operations to generate the final query response, including: All subquery results are obtained from the multi-center network nodes to form a subquery result set; Determine the semantic completeness of each subquery result in the subquery result set. If the semantic completeness is lower than a preset completeness threshold, perform semantic completion. According to the preset association rules, the data objects in the semantically completed subquery result set are integrated to obtain an aggregated data object set; The final query response is generated based on the aggregated data object set.

5. A data multi-center management system based on DOA, characterized in that, The method for implementing the DOA-based multi-center data management method as described in any one of claims 1 to 4 includes: The query decomposition module is used to obtain query requests and data object metadata, and perform semantic decomposition to obtain a set of subquery tasks; The candidate filtering module is used to obtain the computing power indicators and dynamic load data of each data center based on the set of subquery tasks, and to filter and determine the candidate data center list. The plan generation module is used to generate an initial execution plan based on the candidate data center list by comprehensively evaluating network latency and heterogeneous computing capabilities. The execution monitoring module is used to execute the set of sub-query tasks according to the initial execution plan, and to monitor the execution process in real time and identify transmission bottlenecks. The dynamic adjustment module is used to dynamically adjust the initial execution plan and generate an optimized execution plan if the transmission bottleneck is identified. The result aggregation module is used to integrate all subquery results obtained after executing the initial execution plan or the optimized execution plan, and process them using aggregation operations to generate the final query response.

Images