CDC synchronization method and system based on Oracle RAC

Abstract

The present invention provides a CDC synchronization method and system based on Oracle RAC, which relates to the field of Oracle RAC technology, including deploying an incremental log parser, a session state collector and a resource monitor to respectively collect atomic operations, concurrent transaction information and node performance indicators. An intelligent classification model is used to identify data flow characteristics, a priority matrix is ​​constructed, and high-priority data is fine-grainedly collected based on an adaptive sampling strategy. The present invention also uses multi-dimensional evaluation indicators to construct a time series prediction model to predict node loads, and allocates data synchronization tasks to low-load nodes based on a load balancing matrix, and adopts a two-phase commit protocol to ensure data consistency. The present invention effectively improves the efficiency and stability of CDC synchronization in an Oracle RAC environment by monitoring the synchronization status in real time and dynamically adjusting the allocation strategy, and reduces synchronization delays.

Description

Technical Field

[0001] The present invention relates to OracleRAC technology, and in particular to a CDC synchronization method and system based on OracleRAC. Background Art

[0002] Change data capture (CDC) technology is widely used in data synchronization, data integration, data analysis and other fields. Its goal is to capture data changes in the database in real time and propagate them to downstream systems. Traditional CDC methods are usually based on mechanisms such as triggers, timestamps or snapshots, but these methods have some limitations.

[0003] Performance loss: The traditional trigger-based CDC method will have a significant impact on database performance, especially in high-concurrency and high-load scenarios. The snapshot method requires regular creation of database snapshots, which will also consume a lot of system resources.

[0004] Data consistency is difficult to ensure: In a complex distributed environment, it is a challenge to ensure the consistency of data synchronization. Traditional CDC methods are difficult to effectively handle concurrent transactions and data conflicts, which may lead to data inconsistency or data loss.

[0005] Lack of adaptive capabilities: Traditional CDC methods usually adopt fixed synchronization strategies and cannot be dynamically adjusted according to system load and data changes. This leads to low synchronization efficiency and even excessive synchronization delay when the frequency of data changes fluctuates greatly or system resources are tight. Summary of the invention

[0006] The embodiments of the present invention provide a CDC synchronization method and system based on Oracle RAC, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a CDC synchronization method based on Oracle RAC, including:

[0009] Deploy an incremental log parser at the database layer to capture atomic operations in Oracle redo logs, deploy a session state collector at the application layer to obtain concurrent transaction information, and deploy a resource monitor at the system layer to collect node performance indicators such as processor usage, memory occupancy, and input and output loads; input the atomic operations, concurrent transaction information, and node performance indicators into an intelligent classification model to identify the change frequency, data volume, and complexity characteristics of data flows; construct a priority matrix based on the change frequency, data volume, and complexity characteristics of data flows to classify data changes in real time; design an adaptive sampling strategy based on the priority matrix to implement fine-grained collection of priority data above a preset adaptive threshold;

[0010] The node performance index is used to construct a multi-dimensional evaluation index, which includes a node load index, a network congestion level, and a disk read and write pressure; the multi-dimensional evaluation index is used as an input of a time series prediction model, the historical change trend of the node performance index is analyzed, and a load prediction value in a future time window is generated; the available resource capacity of each node is calculated according to the load prediction value; the data change events in the real-time data synchronization queue are dynamically allocated based on the available resource capacity, and the data change events are encapsulated as data synchronization tasks; a load balancing matrix is ​​constructed based on the node load index, and the data synchronization tasks are allocated to nodes whose load values ​​are lower than the load threshold;

[0011] Start the distributed transaction coordinator to process the data synchronization tasks in parallel; use the two-phase commit protocol to process the data synchronization tasks to ensure data consistency; monitor the execution status of the data synchronization tasks in real time, record the transmission time of the data from the source to the target to obtain the synchronization delay time, compare the source data with the target data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage; when a synchronization anomaly is detected, put the abnormal data synchronization task back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, trigger the load rebalancing process based on the resource usage, and recalculate the load balancing matrix; use the synchronization delay time, the consistency check result and the resource usage as feedback data to optimize the prediction accuracy of the time series prediction model and dynamically adjust the allocation strategy of the data synchronization task.

[0012] Inputting the atomic operation, the concurrent transaction information and the node performance index into an intelligent classification model to identify data flow characteristics; constructing a priority matrix based on the data flow characteristics to classify data changes in real time includes:

[0013] Inputting the atomic operation, the concurrent transaction information and the node performance index into an intelligent classification model, wherein the intelligent classification model adopts a deep neural network structure; extracting data flow features through the intelligent classification model, wherein the data flow features include timing features, load features and dependency features;

[0014] Calculate a feature vector based on the data stream feature; input the feature vector into an adaptive weight calculation unit to generate a feature weight coefficient; multiply the feature weight coefficient by the feature vector to construct a priority matrix;

[0015] Based on the priority matrix, data changes are graded in real time, and data changes are divided into priority data higher than a preset adaptive threshold and priority data lower than the preset adaptive threshold; fine-grained collection is implemented for the priority data higher than the preset adaptive threshold, and the fine-grained collection includes: establishing a dedicated collection channel, capturing the complete change sequence in real time, and recording related transaction information.

[0016] Designing an adaptive sampling strategy according to the priority matrix and implementing fine-grained collection of priority data above a preset adaptive threshold includes:

[0017] Mapping the priority information in the priority matrix to a preset interval by a maximization-minimization normalization method to obtain a normalized priority value;

[0018] Based on the normalized priority value, an adaptive threshold calculation model is established in the sliding time window, and an exponential smoothing algorithm is used to calculate the adaptive threshold in combination with the mean and standard deviation of the priority value in the current time window, wherein the exponential smoothing algorithm includes a smoothing coefficient and a threshold adjustment coefficient;

[0019] Comparing the normalized priority value with the adaptive threshold, and for priority data higher than the adaptive threshold, dynamically adjusting the sampling frequency based on the difference between the normalized priority value and the adaptive threshold, wherein the sampling frequency increases as the difference increases;

[0020] Constructing a double-ended queue for priority mutation monitoring, setting a priority mutation detection threshold, and adjusting the length of the double-ended queue adaptively with changes in the priority mutation detection threshold;

[0021] Monitor the changes of the normalized priority value in real time. When it is detected that the normalized priority value jumps relative to the priority mutation detection threshold, increase the number of sampling bits from the basic number of bits to a higher number of bits, triggering the high-frequency sampling mode to perform fine-grained collection of priority data higher than the preset adaptive threshold.

[0022] The node performance index is used to construct a multi-dimensional evaluation index, which includes a node load index, a network congestion level, and a disk read / write pressure; the multi-dimensional evaluation index is used as an input of a time series prediction model, the historical change trend of the node performance index is analyzed, and a load prediction value in a future time window is generated; and the available resource capacity of each node is calculated according to the load prediction value, including:

[0023] Constructing multi-dimensional evaluation indicators based on the node performance indicators, calculating the node load index, which is based on a weighted combination of processor utilization and memory pressure; calculating the network congestion level, which is comprehensively evaluated by network throughput, data packet delay and packet loss rate; calculating the disk read and write pressure, which is evaluated in combination with IO response time and storage queue depth;

[0024] Organize the multi-dimensional evaluation indicators in time series to form an indicator sequence; perform data cleaning on the indicator sequence to eliminate noise points and outliers; extract the long-term change trend, periodic fluctuation characteristics and burst mode of the indicator sequence; build a time series prediction model, which adopts a deep neural network architecture; input the long-term change trend, the periodic fluctuation characteristics and the burst mode into the time series prediction model; analyze the historical change rules based on the time series prediction model to generate load prediction values ​​in the future time window;

[0025] According to the load prediction value, the node resource status is evaluated, and the available computing power of the processor is calculated, and the available computing power takes into account the multi-core task scheduling efficiency; the available memory capacity is estimated, and the available memory capacity is based on the page replacement rate and the degree of memory fragmentation; the network bandwidth margin is calculated, and the network bandwidth margin is combined with the link utilization and the transmission queue status; the storage system capacity is evaluated, and the storage system capacity is determined by the IO throughput and cache efficiency; a resource competition model is established to analyze the degree of interference between different load types; and the available resource capacity is adjusted according to the degree of interference.

[0026] Dynamically allocating data change events in the real-time data synchronization queue based on the available resource capacity, encapsulating the data change events as data synchronization tasks; constructing a load balancing matrix based on the node load index, and allocating the data synchronization tasks to nodes whose load values ​​are lower than a load threshold comprises:

[0027] Dynamically allocate data change events in the data synchronization queue according to the available resource capacity, monitor the waiting time of the data change event, and when the waiting time exceeds the preset time, increase the event priority through an aging algorithm; identify continuous change events of the same data table in the data synchronization queue, and integrate the continuous change events into batch tasks; generate data synchronization tasks based on the available resource capacity, the event priority and the batch tasks;

[0028] Construct a load balancing matrix, which includes a node resource status dimension, a synchronization task feature dimension, and a historical execution effect dimension; in the node resource status dimension, record the node's processor usage, memory occupancy, and input and output pressure, and calculate the node's current load value; in the synchronization task feature dimension, record the resource demand and execution priority of the data synchronization task; in the historical execution effect dimension, record the node's task completion rate and response delay;

[0029] The data synchronization task is allocated based on the load balancing matrix, and the node with the lowest current load value and meeting the resource demand is selected as the target node; the data synchronization task is allocated to the target node; the load status of the target node is continuously monitored, and when the current load value of the target node exceeds the load threshold, the task migration mechanism is triggered; the load balancing matrix information of other nodes is re-acquired, and the node with the lowest load is selected as the migration target.

[0030] Start the distributed transaction coordinator to process the data synchronization task in parallel; use the two-phase commit protocol to process the data synchronization task to ensure data consistency; monitor the execution status of the data synchronization task in real time, record the transmission time of data from the source end to the target end to obtain the synchronization delay time, compare the source end data with the target end data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage, including:

[0031] Starting a distributed transaction coordinator, analyzing the dependencies of the data synchronization tasks, and constructing a task dependency graph; identifying a task subset without dependencies based on the task dependency graph, and marking the task subset as a parallel task group; monitoring system resource status, and dynamically calculating the maximum concurrency number based on processor load and memory usage; and allocating tasks to the parallel task group based on the maximum concurrency number;

[0032] Before executing the parallelizable task group, a two-phase commit protocol is started; in the first phase, a prepare request is sent to the participating nodes, the prepare request includes synchronization data content and data table structure information; the participating nodes obtain an exclusive lock on the data table and verify the consistency of the data table structure; the synchronization data is written into a temporary storage area and a data checksum is generated; the participating nodes return a prepare response, the prepare response includes data checksum and resource reservation status; in the second phase, a decision is made whether to commit based on the prepare responses of all participating nodes;

[0033] Collect the execution status of the synchronization process in real time, record the transmission time of data from the source to the target to obtain the synchronization delay time; compare the source data checksum with the target data checksum to generate a consistency check result; collect the resource usage of the participating nodes, including processor usage, memory occupancy and network bandwidth occupancy; calculate performance indicators based on the execution status, and compare the performance indicators with preset performance thresholds; generate performance alarms when the performance indicators exceed the preset performance thresholds.

[0034] When a synchronization anomaly is detected, the abnormal data synchronization task is put back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, the load rebalancing process is triggered based on the resource usage, and the load balancing matrix is ​​recalculated; the synchronization delay time, the consistency check result and the resource usage are used as feedback data to optimize the prediction accuracy of the time series prediction model, and the allocation strategy of the data synchronization task is dynamically adjusted, including:

[0035] When the consistency check result shows that the data is inconsistent, the current synchronization state is recorded to obtain a synchronization exception mark; the data synchronization task with the synchronization exception mark and the resource usage are packaged together and put back into the real-time data synchronization queue; the task priority is calculated according to the synchronization delay time, and the real-time data synchronization queue is prioritized; when the synchronization delay time exceeds the preset time threshold, the load balancing matrix is ​​recalculated based on the resource usage; the load balancing matrix is ​​used to quantify the computing node load level;

[0036] For the nodes whose load levels exceed the preset load threshold, the resource usage is analyzed and a load rebalancing process is started; the target nodes whose load levels are lower than the preset load threshold are identified from the load balancing matrix and a task migration plan is calculated; the synchronization delay time, the consistency check result and the resource usage are integrated into a performance evaluation index; the performance evaluation index is used as feedback data and input into a timing prediction model;

[0037] Utilize the feedback data to analyze the trend of system performance changes and update the parameters of the timing prediction model; predict the system load in the future time window based on the updated timing prediction model; recalculate the load balancing matrix according to the predicted system load; and dynamically adjust the allocation strategy of the data synchronization task based on the updated load balancing matrix.

[0038] According to a second aspect of the embodiments of the present invention,

[0039] Provides Oracle RAC-based CDC synchronization system, including:

[0040] The first unit is used to deploy an incremental log parser at the database layer to capture atomic operations in Oracle redo logs, deploy a session state collector at the application layer to obtain concurrent transaction information, and deploy a resource monitor at the system layer to collect node performance indicators such as processor usage, memory occupancy, and input and output loads; input the atomic operations, the concurrent transaction information, and the node performance indicators into an intelligent classification model to identify the change frequency, data volume, and complexity characteristics of the data flow; construct a priority matrix based on the change frequency, data volume, and complexity characteristics of the data flow, and perform real-time classification of data changes; design an adaptive sampling strategy based on the priority matrix, and implement fine-grained collection of priority data above a preset adaptive threshold;

[0041] The second unit is used to construct a multi-dimensional evaluation index using the node performance index, and the multi-dimensional evaluation index includes a node load index, a network congestion level, and a disk read and write pressure; the multi-dimensional evaluation index is used as an input of a time series prediction model, the historical change trend of the node performance index is analyzed, and a load prediction value in a future time window is generated; the available resource capacity of each node is calculated according to the load prediction value; the data change events in the real-time data synchronization queue are dynamically allocated based on the available resource capacity, and the data change events are encapsulated as data synchronization tasks; a load balancing matrix is ​​constructed based on the node load index, and the data synchronization tasks are allocated to nodes whose load values ​​are lower than a load threshold;

[0042] The third unit is used to start the distributed transaction coordinator to process the data synchronization tasks in parallel; use the two-phase commit protocol to process the data synchronization tasks to ensure data consistency; monitor the execution status of the data synchronization tasks in real time, record the transmission time of the data from the source to the target to obtain the synchronization delay time, compare the source data with the target data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage; when a synchronization anomaly is detected, the abnormal data synchronization task is put back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, the load rebalancing process is triggered based on the resource usage, and the load balancing matrix is ​​recalculated; the synchronization delay time, the consistency check result and the resource usage are used as feedback data to optimize the prediction accuracy of the time series prediction model and dynamically adjust the allocation strategy of the data synchronization task.

[0043] According to a third aspect of the embodiments of the present invention,

[0044] An electronic device is provided, comprising:

[0045] processor;

[0046] a memory for storing processor-executable instructions;

[0047] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0048] According to a fourth aspect of the embodiments of the present invention,

[0049] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0050] The beneficial effects of this application are as follows:

[0051] 1. Improve data synchronization efficiency: The intelligent classification model is used to identify data stream characteristics, and combined with an adaptive sampling strategy, data of different priorities are processed in a hierarchical manner, avoiding indiscriminate synchronization of all data, thereby improving data synchronization efficiency and reducing unnecessary resource consumption.

[0052] 2. Ensure the stability and reliability of data synchronization: Use the time series prediction model to predict node performance indicators and perform load balancing based on the prediction results, which can effectively avoid single-point overload and improve system stability. The two-phase commit protocol and distributed transaction coordinator ensure the consistency and reliability of data synchronization. Even if synchronization anomalies occur, they can be handled in a timely manner to ensure that the data is eventually consistent.

[0053] 3. Dynamic allocation and optimization of resources: Dynamic allocation of resources based on node performance indicators and load prediction values ​​can fully utilize system resources and avoid resource waste. By real-time monitoring of synchronization delay time, consistency check results, and resource usage, and using them as feedback data to optimize the timing prediction model and data synchronization task allocation strategy, dynamic optimization and adaptive adjustment of resources are achieved, further improving system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of a CDC synchronization method based on Oracle RAC according to an embodiment of the present invention;

[0055] Figure 2 It is a structural diagram of a CDC synchronization system based on Oracle RAC according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0058] Figure 1 FIG. 1 is a flow chart of a CDC synchronization method based on Oracle RAC according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0059] S11. Deploy an incremental log parser at the database layer to capture atomic operations in Oracle redo logs, deploy a session state collector at the application layer to obtain concurrent transaction information, and deploy a resource monitor at the system layer to collect node performance indicators such as processor usage, memory occupancy, and input and output loads; input the atomic operations, the concurrent transaction information, and the node performance indicators into an intelligent classification model to identify the change frequency, data volume, and complexity characteristics of the data flow; construct a priority matrix based on the change frequency, data volume, and complexity characteristics of the data flow, and perform real-time classification of data changes; design an adaptive sampling strategy based on the priority matrix, and implement fine-grained collection of priority data above a preset adaptive threshold;

[0060] S12. Use the node performance indicators to construct multi-dimensional evaluation indicators, which include node load index, network congestion level and disk read and write pressure; use the multi-dimensional evaluation indicators as input of the time series prediction model, analyze the historical change trend of the node performance indicators, and generate load prediction values ​​in the future time window; calculate the available resource capacity of each node according to the load prediction value; dynamically allocate data change events in the real-time data synchronization queue based on the available resource capacity, and encapsulate the data change events into data synchronization tasks; construct a load balancing matrix based on the node load index, and allocate the data synchronization tasks to nodes whose load values ​​are lower than the load threshold;

[0061] S13. Start the distributed transaction coordinator to process the data synchronization tasks in parallel; use the two-phase commit protocol to process the data synchronization tasks to ensure data consistency; monitor the execution status of the data synchronization tasks in real time, record the transmission time of the data from the source to the target to obtain the synchronization delay time, compare the source data with the target data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage; when a synchronization anomaly is detected, put the abnormal data synchronization task back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, trigger the load rebalancing process based on the resource usage, and recalculate the load balancing matrix; use the synchronization delay time, the consistency check result and the resource usage as feedback data to optimize the prediction accuracy of the time series prediction model and dynamically adjust the allocation strategy of the data synchronization task.

[0062] In an optional implementation, the atomic operation, the concurrent transaction information, and the node performance index are input into an intelligent classification model to identify data flow characteristics; a priority matrix is ​​constructed based on the data flow characteristics to perform real-time classification of data changes, including:

[0063] Inputting the atomic operation, the concurrent transaction information and the node performance index into an intelligent classification model, wherein the intelligent classification model adopts a deep neural network structure; extracting data flow features through the intelligent classification model, wherein the data flow features include timing features, load features and dependency features;

[0064] Calculate a feature vector based on the data stream feature; input the feature vector into an adaptive weight calculation unit to generate a feature weight coefficient; multiply the feature weight coefficient by the feature vector to construct a priority matrix;

[0065] Based on the priority matrix, data changes are graded in real time, and data changes are divided into priority data higher than a preset adaptive threshold and priority data lower than the preset adaptive threshold; fine-grained collection is implemented for the priority data higher than the preset adaptive threshold, and the fine-grained collection includes: establishing a dedicated collection channel, capturing the complete change sequence in real time, and recording related transaction information.

[0066] The invention discloses a real-time classification method and system for data changes, which are used to classify and collect data changes in a database or other system in real time, thereby improving system performance and data analysis efficiency.

[0067] First, collect atomic operations, concurrent transaction information, and node performance indicators in the system. Atomic operations refer to the smallest operation unit in the database, such as inserting, deleting, and updating a record. Concurrent transaction information refers to the concurrent execution of multiple transactions in the same time period, including the start time, end time, and data items involved in the transaction. Node performance indicators refer to the performance indicators of database server nodes, such as CPU usage, memory usage, and disk I / O. For example, the atomic operation collected is "update the value of field B of record ID=1 in table A to 2", the concurrent transaction information is "transactions T1 and T2 update table A at the same time", and the node performance indicator is "CPU usage 80%".

[0068] Then, the collected data is input into the intelligent classification model. The model uses a deep neural network structure, such as a multi-layer perceptron, a convolutional neural network, etc. After training, the model can identify the timing characteristics, load characteristics, and dependency characteristics in the data stream. The timing characteristics refer to the time pattern of data changes, such as periodic changes, sudden changes, etc. The load characteristics refer to the occupation of system resources by data changes, such as CPU load, memory load, etc. The dependency characteristics refer to the dependency relationship between different data changes, such as the dependency relationship between transactions, the dependency relationship between data tables, etc. For example, through model analysis, the data stream characteristics extracted are "timing characteristics: periodic changes, data updates at 12 o'clock every night; load characteristics: high CPU load, CPU usage reaches 80% when updating data; dependency characteristics: there is a dependency relationship between transactions T1 and T2, T1 must be executed before T2".

[0069] Next, the feature vector is calculated based on the extracted data stream features. Each feature is quantified into a numerical value to form a feature vector. For example, if "periodic change" is quantified to 1, "high CPU load" is quantified to 0.8, and "transaction dependency" is quantified to 0.5, the feature vector is [1, 0.8, 0.5]. Then, the feature vector is input into the adaptive weight calculation unit to generate the feature weight coefficient. The unit dynamically adjusts the weight coefficient of each feature based on the feature vector and historical data. For example, based on the analysis of historical data, it is found that "high CPU load" has a greater impact on system performance, so the weight coefficient of "high CPU load" is set to 0.9, and the weight coefficients of other features are set to 0.3 and 0.2 respectively.

[0070] Afterwards, the feature weight coefficients are multiplied by the feature vectors to construct the priority matrix.

[0071] Finally, data changes are graded in real time based on the priority matrix. A preset adaptive threshold is set, such as 0.8. Data changes with priority values ​​higher than the threshold are classified as high priority data, and data changes below the threshold are classified as low priority data. For example, if the priority value 1.12 is higher than the threshold 0.8, the data change is classified as high priority data. For high-priority data, fine-grained collection is implemented. Establish a dedicated collection channel to capture the complete change sequence in real time and record the associated transaction information. For example, for the high-priority data change "update the field B value of record ID=1 of table A to 2", record its complete change sequence, including the transaction start time, end time, updated field value, etc., as well as the information of associated transactions T1 and T2.

[0072] The solution of this application can:

[0073] Improve system performance: By collecting high-priority data changes in a fine-grained manner, the collection and processing overhead of low-priority data changes can be reduced, thereby improving the overall performance of the system. Optimize data analysis: Fine-grained data collection contains more complete change information, which can provide more accurate data support for data analysis and improve the efficiency and accuracy of data analysis. Enhance system stability: By identifying high-priority data changes and conducting real-time monitoring, potential system risks can be discovered in a timely manner, and corresponding measures can be taken to enhance system stability.

[0074] In an optional implementation, designing an adaptive sampling strategy according to the priority matrix and implementing fine-grained collection of priority data above a preset adaptive threshold includes:

[0075] Mapping the priority information in the priority matrix to a preset interval by a maximization-minimization normalization method to obtain a normalized priority value;

[0076] Based on the normalized priority value, an adaptive threshold calculation model is established in the sliding time window, and an exponential smoothing algorithm is used to calculate the adaptive threshold in combination with the mean and standard deviation of the priority value in the current time window, wherein the exponential smoothing algorithm includes a smoothing coefficient and a threshold adjustment coefficient;

[0077] Comparing the normalized priority value with the adaptive threshold, and for priority data higher than the adaptive threshold, dynamically adjusting the sampling frequency based on the difference between the normalized priority value and the adaptive threshold, wherein the sampling frequency increases as the difference increases;

[0078] Constructing a double-ended queue for priority mutation monitoring, setting a priority mutation detection threshold, and adjusting the length of the double-ended queue adaptively with changes in the priority mutation detection threshold;

[0079] Monitor the changes of the normalized priority value in real time. When it is detected that the normalized priority value jumps relative to the priority mutation detection threshold, increase the number of sampling bits from the basic number of bits to a higher number of bits, triggering the high-frequency sampling mode to perform fine-grained collection of priority data higher than the preset adaptive threshold.

[0080] An adaptive sampling method based on priority matrix is ​​used to collect important data in fine-grained manner. The core idea of ​​this method is to dynamically adjust the sampling frequency according to the priority of the data. The higher the priority, the higher the sampling frequency, so as to collect as much important data as possible under limited resources.

[0081] First, normalize the priority information in the priority matrix. Assume that the value range in the priority matrix is ​​between 1 and 10, and the target interval is between 0 and 1. Use the maximum-minimum normalization method to subtract the minimum value 1 from each priority value, and then divide it by the difference between the maximum and minimum values ​​9 to obtain the normalized priority value, which ranges from 0 to 1. For example, the priority value is 5, and the normalized value is (5-1) / (10-1)=4 / 9≈0.44.

[0082] Next, an adaptive threshold calculation model is established within the sliding time window. Assume that the time window size is 10 seconds. The adaptive threshold is calculated using an exponential smoothing algorithm. The algorithm combines the mean and standard deviation of the priority value within the current time window and introduces a smoothing coefficient and a threshold adjustment coefficient. The smoothing coefficient is used to control the degree of influence of historical data on the current threshold, and the threshold adjustment coefficient is used to adjust the sensitivity of the threshold.

[0083] Then, the normalized priority value is compared with the adaptive threshold. For priority data higher than the adaptive threshold, the sampling frequency is dynamically adjusted according to the difference between the normalized priority value and the adaptive threshold. The larger the difference, the higher the sampling frequency. For example, assuming the base sampling frequency is 1Hz, the priority value is 8, the normalized priority value is (8-1) / (10-1)=7 / 9≈0.78, and the adaptive threshold is 0.6, then the difference is 0.78-0.6=0.18. Assuming that the sampling frequency increases linearly with the difference, with a slope of 10, the current sampling frequency is 1+10*0.18=2.8Hz.

[0084] At the same time, a double-ended queue is constructed for priority mutation monitoring. The priority mutation detection threshold is set. The length of the double-ended queue is adaptively adjusted according to the change of the priority mutation detection threshold. For example, the priority mutation detection threshold is 0.2, and the initial value of the queue length is 5. When it is detected that the normalized priority value has jumped relative to the priority mutation detection threshold, for example, it suddenly rises from 0.5 to 0.8, exceeding the mutation detection threshold of 0.2, the sampling bit number is increased from the basic bit number (for example, 8 bits) to a higher bit number (for example, 16 bits), triggering the high-frequency sampling mode, and performing fine-grained collection of priority data above the adaptive threshold. If the mutation detection threshold is adjusted to 0.3, the queue length is adjusted to 7 accordingly to adapt to the new threshold.

[0085] The solution of this application can:

[0086] Improve data collection efficiency: This method can dynamically adjust the sampling frequency according to the priority of the data, and concentrate limited resources on collecting high-priority data, thereby improving data collection efficiency and avoiding resource waste. Ensure the integrity of key data: By performing fine-grained collection of data above the adaptive threshold, and real-time monitoring and high-frequency sampling of priority mutations, the integrity and accuracy of key data can be effectively guaranteed, and important information can be avoided from being lost. Enhance system adaptability: The adaptive threshold and the adaptive adjustment mechanism of the double-ended queue length enable this method to flexibly adapt to different data environments and application scenarios, improving the robustness and adaptability of the system.

[0087] In an optional implementation, the node performance index is used to construct a multi-dimensional evaluation index, and the multi-dimensional evaluation index includes a node load index, a network congestion level, and a disk read and write pressure; the multi-dimensional evaluation index is used as an input of a time series prediction model, and the historical change trend of the node performance index is analyzed to generate a load prediction value in a future time window; and the available resource capacity of each node is calculated according to the load prediction value, including:

[0088] Constructing multi-dimensional evaluation indicators based on the node performance indicators, calculating the node load index, which is based on a weighted combination of processor utilization and memory pressure; calculating the network congestion level, which is comprehensively evaluated by network throughput, data packet delay and packet loss rate; calculating the disk read and write pressure, which is evaluated in combination with IO response time and storage queue depth;

[0089] Organize the multi-dimensional evaluation indicators in time series to form an indicator sequence; perform data cleaning on the indicator sequence to eliminate noise points and outliers; extract the long-term change trend, periodic fluctuation characteristics and burst mode of the indicator sequence; build a time series prediction model, which adopts a deep neural network architecture; input the long-term change trend, the periodic fluctuation characteristics and the burst mode into the time series prediction model; analyze the historical change rules based on the time series prediction model to generate load prediction values ​​in the future time window;

[0090] According to the load prediction value, the node resource status is evaluated, and the available computing power of the processor is calculated, and the available computing power takes into account the multi-core task scheduling efficiency; the available memory capacity is estimated, and the available memory capacity is based on the page replacement rate and the degree of memory fragmentation; the network bandwidth margin is calculated, and the network bandwidth margin is combined with the link utilization and the transmission queue status; the storage system capacity is evaluated, and the storage system capacity is determined by the IO throughput and cache efficiency; a resource competition model is established to analyze the degree of interference between different load types; and the available resource capacity is adjusted according to the degree of interference.

[0091] A node resource capacity prediction method based on multi-dimensional evaluation indicators is used to accurately predict the available resource capacity of nodes in a future time window.

[0092] First, construct a multi-dimensional evaluation index. Collect the processor utilization and memory pressure values ​​of the node, set the weights according to historical data and experience, for example, the processor utilization weight is 0.6, the memory pressure weight is 0.4, and the weighted combination of the two is calculated to obtain the node load index. For example, if the processor utilization is 70% and the memory pressure is 80%, the node load index is 70%*0.6+80%*0.4=74. At the same time, collect network throughput, packet delay and packet loss rate, for example, the throughput is 90Mbps, the delay is 20ms, and the packet loss rate is 1%. According to the preset evaluation rules, for example, the higher the throughput, the lower the delay, and the lower the packet loss rate, the lower the network congestion level, and comprehensively evaluate the network congestion level. In addition, collect disk IO response time and storage queue depth, for example, the response time is 10ms and the queue depth is 5, and combine the two to evaluate the disk read and write pressure. Assuming that the shorter the response time and the shallower the queue depth, the lower the disk read and write pressure.

[0093] Next, the multi-dimensional evaluation indicators such as node load index, network congestion level, and disk read and write pressure are organized into indicator sequences in chronological order. Data cleaning is performed on the indicator sequence, such as using the moving average method to smooth the data and remove noise points and outliers. For example, if the network throughput at a certain time point suddenly drops to close to 0, and the data deviation from the previous and next time points is large, it is judged as an outlier and the average value of the previous and next time points is used to replace the outlier. Then, the long-term change trend of the indicator sequence is extracted, such as using linear regression analysis methods to fit long-term trend lines, identifying the periodic fluctuation characteristics of the indicator sequence, such as using Fourier transform analysis methods to identify periodic fluctuations, and identifying burst patterns, such as using peak detection algorithms to identify burst peaks and valleys.

[0094] Then, build a time series prediction model. Use a deep neural network architecture, such as a long short-term memory network (LSTM) or a gated recurrent unit (GRU) network, and extract long-term change trends, periodic fluctuation characteristics, and burst patterns as input features. Use historical data to train the time series prediction model. For example, use the node performance indicator data of the past week as a training set to predict the load of the next day. Through model training, learn the changing patterns of historical data.

[0095] Next, based on the trained time series prediction model, the load prediction value in the future time window is generated. For example, the node load index, network congestion level, and disk read and write pressure are predicted for each hour in the next 24 hours.

[0096] Finally, the available resource capacity of each node is calculated based on the load prediction value. For example, based on the predicted processor utilization, the available computing power of the processor is calculated considering the multi-core task scheduling efficiency. Assuming that the predicted processor utilization is 80%, the single-core efficiency is 90%, and the dual-core efficiency is 85%, the available computing power is estimated based on the task scheduling strategy, such as giving priority to the dual-core. According to the predicted memory pressure, the available memory capacity is estimated based on the page replacement rate and the degree of memory fragmentation. For example, if the predicted memory pressure is high and the page replacement rate increases, the available memory capacity is reduced accordingly. According to the predicted network congestion level, the network bandwidth margin is calculated in combination with the link utilization and the transmission queue status. For example, if the predicted network congestion level is high, the available bandwidth margin is reduced. According to the predicted disk read and write pressure, the storage system capacity is determined by the IO throughput and cache efficiency. For example, if the predicted disk read and write pressure is high, the available storage capacity is reduced. A resource competition model is established to analyze the degree of interference between different load types. For example, high IO load will affect the performance of the CPU, and the available CPU computing power is adjusted according to the degree of interference.

[0097] The solution of this application can:

[0098] Improve resource utilization: By accurately predicting the available resource capacity of nodes, resources can be allocated and utilized more efficiently, avoiding resource waste and maximizing resource utilization. Enhance system stability: By predicting future load conditions, potential resource bottlenecks and risks can be identified in advance, and corresponding measures can be taken, such as dynamically adjusting resource allocation or adding nodes, thereby improving system stability and reliability. Optimize user experience: By predicting load, resources can be reserved in advance to ensure the quality of service, reduce the delay and response time of user requests, and thus improve user experience.

[0099] In an optional implementation, dynamically allocating data change events in the real-time data synchronization queue based on the available resource capacity, encapsulating the data change events as data synchronization tasks; constructing a load balancing matrix based on the node load index, and allocating the data synchronization tasks to nodes whose load values ​​are lower than a load threshold includes:

[0100] Dynamically allocate data change events in the data synchronization queue according to the available resource capacity, monitor the waiting time of the data change event, and when the waiting time exceeds the preset time, increase the event priority through an aging algorithm; identify continuous change events of the same data table in the data synchronization queue, and integrate the continuous change events into batch tasks; generate data synchronization tasks based on the available resource capacity, the event priority and the batch tasks;

[0101] Construct a load balancing matrix, which includes a node resource status dimension, a synchronization task feature dimension, and a historical execution effect dimension; in the node resource status dimension, record the node's processor usage, memory occupancy, and input and output pressure, and calculate the node's current load value; in the synchronization task feature dimension, record the resource demand and execution priority of the data synchronization task; in the historical execution effect dimension, record the node's task completion rate and response delay;

[0102] The data synchronization task is allocated based on the load balancing matrix, and the node with the lowest current load value and meeting the resource demand is selected as the target node; the data synchronization task is allocated to the target node; the load status of the target node is continuously monitored, and when the current load value of the target node exceeds the load threshold, the task migration mechanism is triggered; the load balancing matrix information of other nodes is re-acquired, and the node with the lowest load is selected as the migration target.

[0103] In real-time data synchronization systems, efficient processing of data change events and load balancing are crucial. The following describes a dynamic task allocation method based on available resource capacity and load balancing matrix to optimize data synchronization efficiency and system stability.

[0104] First, the system dynamically allocates data change events in the data synchronization queue. The system monitors the available resource capacity in real time, such as CPU, memory, and network bandwidth. Based on the available resource capacity, the system determines the number of data change events that can be processed. At the same time, the system monitors the waiting time of each data change event in the queue. If the waiting time of an event exceeds the preset time, such as 3 seconds, its priority is increased through the aging algorithm, such as from low to medium, to ensure that important events are processed in a timely manner. In addition, the system identifies continuous change events of the same data table in the data synchronization queue, such as continuous update operations on the user table. These continuous change events are integrated into batch tasks, such as merging 10 update operations on the user table into one batch update task, to reduce the number of database interactions and improve processing efficiency. Finally, the system generates data synchronization tasks based on the available resource capacity, event priority, and batch tasks. For example, if the current available resource capacity is sufficient and there are high-priority events and batch tasks, high-priority data synchronization tasks and batch tasks are generated first.

[0105] Next, the system builds a load balancing matrix to guide the allocation of data synchronization tasks. The matrix contains the node resource status dimension, the synchronization task feature dimension, and the historical execution effect dimension. In the node resource status dimension, the system records the processor usage, memory occupancy, and input and output pressure of each node, and calculates the current load value of the node based on these indicators. For example, if the CPU usage of node A is 70%, the memory occupancy is 80%, and the input and output pressure is 50%, then its load value is relatively high. In the synchronization task feature dimension, the system records the resource demand and execution priority of each data synchronization task. For example, a task that batch updates user table data has a high resource demand and a high priority. In the historical execution effect dimension, the system records the task completion rate and response delay of each node. For example, if the historical task completion rate of node B is 95% and the average response delay is 100 milliseconds, then its historical execution effect is relatively good.

[0106] Then, the system allocates data synchronization tasks based on the load balancing matrix. The system selects the node with the lowest current load value and meets the task resource requirements as the target node. For example, if a data synchronization task requires 10% of CPU resources and 20% of memory resources, and the priority is medium, the system will select the node with low CPU usage and memory occupancy and can meet the resource requirements as the target node. After the data synchronization task is assigned to the target node, the system continuously monitors the load status of the target node. When the current load value of the target node exceeds the load threshold, such as 90%, the task migration mechanism is triggered. The system re-acquires the load balancing matrix information of other nodes, selects the node with the lowest load as the migration target, and migrates the task to this node to balance the load and avoid single point overload.

[0107] The solution of this application can:

[0108] Improve data synchronization efficiency: By increasing the priority of events with long waiting times through the aging algorithm, high-priority tasks and batch tasks are prioritized, and continuous change events are integrated into batch tasks, which can effectively reduce data synchronization delays and improve data synchronization efficiency. Optimize resource utilization: Dynamically allocate data change events based on available resource capacity, and assign tasks to nodes with lower loads according to the load balancing matrix, which can effectively avoid resource waste and improve resource utilization. Enhance system stability: By continuously monitoring the node load status and triggering the task migration mechanism when the load is too high, single point overload can be effectively avoided, and system stability and fault tolerance can be enhanced.

[0109] In an optional implementation, a distributed transaction coordinator is started to process the data synchronization task in parallel; a two-phase commit protocol is used to process the data synchronization task to ensure data consistency; the execution status of the data synchronization task is monitored in real time, the transmission time of data from the source end to the target end is recorded to obtain the synchronization delay time, the source end data is compared with the target end data to obtain the consistency check result, and the node performance index is collected to obtain the resource usage, including:

[0110] Starting a distributed transaction coordinator, analyzing the dependencies of the data synchronization tasks, and constructing a task dependency graph; identifying a task subset without dependencies based on the task dependency graph, and marking the task subset as a parallel task group; monitoring system resource status, and dynamically calculating the maximum concurrency number based on processor load and memory usage; and allocating tasks to the parallel task group based on the maximum concurrency number;

[0111] Before executing the parallelizable task group, a two-phase commit protocol is started; in the first phase, a prepare request is sent to the participating nodes, the prepare request includes synchronization data content and data table structure information; the participating nodes obtain an exclusive lock on the data table and verify the consistency of the data table structure; the synchronization data is written into a temporary storage area and a data checksum is generated; the participating nodes return a prepare response, the prepare response includes data checksum and resource reservation status; in the second phase, a decision is made whether to commit based on the prepare responses of all participating nodes;

[0112] Collect the execution status of the synchronization process in real time, record the transmission time of data from the source to the target to obtain the synchronization delay time; compare the source data checksum with the target data checksum to generate a consistency check result; collect the resource usage of the participating nodes, including processor usage, memory occupancy and network bandwidth occupancy; calculate performance indicators based on the execution status, and compare the performance indicators with preset performance thresholds; generate performance alarms when the performance indicators exceed the preset performance thresholds.

[0113] A data synchronization method based on a distributed transaction coordinator is used to achieve efficient and reliable data synchronization, and its specific implementation method is as follows:

[0114] First, start the distributed transaction coordinator to analyze the data synchronization tasks. For example, an e-commerce platform needs to synchronize order data from the order database to the data warehouse, and from the order database to the logistics system. The distributed transaction coordinator analyzes the dependencies of these tasks and builds a task dependency graph. For example, the task of synchronizing to the data warehouse depends on the task of synchronizing to the logistics system, because the data warehouse needs to contain complete order information, including logistics information. The coordinator represents these dependencies in the form of a graph, where nodes represent tasks and edges represent dependencies.

[0115] Next, identify the subset of tasks without dependencies based on the task dependency graph. For example, suppose there are three tasks: A. Synchronize order data to the data warehouse; B. Synchronize order data to the logistics system; C. Synchronize order data to the recommendation system. A depends on B, and C does not depend on A and B. Then mark C as a parallelizable task group.

[0116] Then, the system resource status is monitored, and the maximum concurrency is dynamically calculated based on the processor load and memory usage. For example, assuming that the current system processor load is 50% and the memory usage is 60%, the maximum concurrency is calculated to be 4 based on the preset resource utilization threshold.

[0117] Tasks are assigned to the parallelizable task group according to the maximum concurrency. For example, if the current parallelizable task group contains 4 tasks and the maximum concurrency is 4, these 4 tasks are assigned to 4 different working nodes for execution.

[0118] Before executing the parallelizable task group, start the two-phase commit protocol. In the first phase, send a prepare request to the participating nodes. For example, send a prepare request to the node responsible for synchronizing order data to the recommendation system, which contains the order data to be synchronized (for example, order ID, product ID, order time, etc.) and the structure information of the target data table (for example, field name, data type, etc.). The participating node obtains an exclusive lock on the data table and verifies the consistency of the data table structure. For example, check whether the required fields exist in the target data table and whether the data type of the field is consistent with the source data. Write the synchronized data to the temporary storage area and generate a data checksum, for example, using the MD5 algorithm to calculate the checksum. The participating node returns a prepare response, which contains the data checksum and resource reservation status. For example, "Checksum: ABCD1234, resource reservation successful" is returned.

[0119] In the second phase, the decision on whether to commit is made based on the preparation responses of all participating nodes. For example, if all participating nodes return "resource reservation successful", the coordinator sends a commit instruction; if any node returns "resource reservation failed", the coordinator sends a rollback instruction.

[0120] Collect the execution status of the synchronization process in real time, record the transmission time of data from the source end to the target end, and obtain the synchronization delay time. For example, record the time for data to be transmitted from the order database to the recommendation system, for example, it takes 100 milliseconds. Compare the checksum of the source data with the checksum of the target data to generate a consistency check result. For example, compare the checksum of the data in the order database with the checksum of the data in the recommendation system. If they are consistent, the consistency check result is "consistent". Collect the resource usage of the participating nodes, including the processor usage, memory usage, and network bandwidth usage. For example, collect the processor usage, memory usage, and network bandwidth usage of the recommendation system node during the synchronization process. Calculate performance indicators based on the execution status, such as throughput (the number of orders synchronized per second) and synchronization delay. Compare the performance indicators with the preset performance thresholds, for example, the throughput threshold is 1,000 orders per second and the delay threshold is 200 milliseconds. Generate a performance alarm when the performance indicator exceeds the preset performance threshold. For example, if the synchronization delay exceeds 200 milliseconds, an alarm message "Synchronization delay is too high" is generated.

[0121] The solution of this application can:

[0122] Improve data synchronization efficiency: Through task dependency analysis and parallel processing, maximize the use of system resources and improve data synchronization efficiency. Ensure data consistency: Through the two-phase commit protocol, ensure data consistency between multiple nodes, and ensure data integrity and accuracy even in the event of a failure. Real-time monitoring and alarm: By real-time monitoring of the execution status and resource usage of the synchronization process, and comparing with the preset threshold, timely discover performance bottlenecks and potential problems, and generate alarm information to facilitate administrators to handle in a timely manner, ensuring the stability and reliability of data synchronization.

[0123] In an optional implementation, when a synchronization anomaly is detected, the abnormal data synchronization task is put back into the real-time data synchronization queue; when the synchronization delay time exceeds a preset threshold, the load rebalancing process is triggered based on the resource usage, and the load balancing matrix is ​​recalculated; the synchronization delay time, the consistency check result and the resource usage are used as feedback data to optimize the prediction accuracy of the time series prediction model, and the allocation strategy of the data synchronization task is dynamically adjusted, including:

[0124] When the consistency check result shows that the data is inconsistent, the current synchronization state is recorded to obtain a synchronization exception mark; the data synchronization task with the synchronization exception mark and the resource usage are packaged together and put back into the real-time data synchronization queue; the task priority is calculated according to the synchronization delay time, and the real-time data synchronization queue is prioritized; when the synchronization delay time exceeds the preset time threshold, the load balancing matrix is ​​recalculated based on the resource usage; the load balancing matrix is ​​used to quantify the computing node load level;

[0125] For the nodes whose load levels exceed the preset load threshold, the resource usage is analyzed and a load rebalancing process is started; the target nodes whose load levels are lower than the preset load threshold are identified from the load balancing matrix and a task migration plan is calculated; the synchronization delay time, the consistency check result and the resource usage are integrated into a performance evaluation index; the performance evaluation index is used as feedback data and input into a timing prediction model;

[0126] Utilize the feedback data to analyze the trend of system performance changes and update the parameters of the timing prediction model; predict the system load in the future time window based on the updated timing prediction model; recalculate the load balancing matrix according to the predicted system load; and dynamically adjust the allocation strategy of the data synchronization task based on the updated load balancing matrix.

[0127] A real-time data synchronization method and system are used to solve the data synchronization delay and consistency problems in a distributed environment.

[0128] First, the system is initialized. A real-time data synchronization queue is established to store data tasks to be synchronized. An initial load balancing matrix is ​​constructed, which describes the load capacity and resource usage of each computing node, such as node CPU, memory, network bandwidth, etc. A time series prediction model is created to predict the system load in the future time window. The input parameters of the model include historical synchronization delay, consistency check results, and resource usage. A preset synchronization delay threshold and node load threshold are set to trigger the load rebalancing process.

[0129] Next, perform data synchronization and consistency check. The data synchronization task obtains data changes from the data source and applies the changes to the target node. For example, the data source is database A and the target node is database B. The synchronization task synchronizes the new data in A to B. After the synchronization is completed, perform a consistency check, for example, compare the hash values ​​of the data in A and B to ensure data consistency.

[0130] If the consistency check result shows that the data is inconsistent, the current synchronization status is recorded and a synchronization exception mark is generated, for example, the exception time, data content, error information, etc. The data synchronization task with the synchronization exception mark and the resource usage at that time, such as CPU usage, memory occupancy, etc., are packaged together and put back into the real-time data synchronization queue. The task priority is calculated based on the synchronization delay time. For example, the longer the delay time, the higher the priority. Prioritize the real-time data synchronization queue, for example, using a priority queue.

[0131] If the synchronization delay time exceeds the preset threshold, for example, more than 10 seconds, the load balancing matrix is ​​recalculated based on the current resource usage. For example, if the CPU usage of node A is 90%, the memory usage is 80%, and the CPU usage of node B is 20%, and the memory usage is 30%, the load capacity of nodes A and B is recalculated, the load weight of A is reduced, and the load weight of B is increased.

[0132] For nodes whose load levels exceed the preset load threshold (e.g. 80%), analyze their resource usage, such as CPU, memory, network bandwidth, etc., and start the load rebalancing process. Identify the target nodes whose load levels are lower than the preset load threshold (e.g. 40%) from the load balancing matrix and calculate the task migration plan. For example, migrate part of the synchronization tasks on node A to node B to balance the load.

[0133] Integrate synchronization delay time, consistency check results, and resource usage into performance evaluation indicators. Use the performance evaluation indicators as feedback data and input them into the time series prediction model. Use the feedback data to analyze the performance change trend of the system. For example, use machine learning algorithms to analyze historical data and predict future trends. Update the parameters of the time series prediction model, for example, adjust the model weights or learning rate.

[0134] Predict the system load in the future time window based on the updated time series prediction model, for example, predict the CPU, memory, and network bandwidth usage of each node in the next hour. Recalculate the load balancing matrix based on the predicted system load, for example, adjust the node load weight based on the predicted resource usage. Dynamically adjust the data synchronization task allocation strategy based on the updated load balancing matrix, for example, assign new synchronization tasks to nodes with lower loads.

[0135] Data case: Assume that there are two nodes A and B, and the initial load balancing matrix is ​​A with a weight of 0.5 and B with a weight of 0.5. The delay time of a synchronization task is 15 seconds, which exceeds the preset threshold of 10 seconds, and the consistency check result is inconsistent. At this time, the CPU usage of node A is 90%, and the CPU usage of B is 20%. Recalculate the load balancing matrix, adjust A's weight to 0.3, and B's weight to 0.7. Put the synchronization task back into the queue and assign it to node B first.

[0136] The solution of this application can:

[0137] Improve data synchronization efficiency: By dynamically adjusting the allocation strategy of data synchronization tasks and load rebalancing, data synchronization delay can be effectively reduced and data synchronization efficiency can be improved. Ensure data consistency: Through consistency verification and synchronization exception handling mechanisms, data inconsistencies can be discovered and corrected in a timely manner to ensure final data consistency. Optimize resource utilization: Through load balancing and timing prediction models, system resources can be effectively allocated and utilized, avoiding resource waste and bottlenecks, and improving overall system performance.

[0138] Figure 2 FIG. 1 is a schematic diagram of the structure of a CDC synchronization system based on Oracle RAC according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0139] The first unit is used to deploy an incremental log parser at the database layer to capture atomic operations in Oracle redo logs, deploy a session state collector at the application layer to obtain concurrent transaction information, and deploy a resource monitor at the system layer to collect node performance indicators such as processor usage, memory occupancy, and input and output loads; input the atomic operations, the concurrent transaction information, and the node performance indicators into an intelligent classification model to identify the change frequency, data volume, and complexity characteristics of the data flow; construct a priority matrix based on the change frequency, data volume, and complexity characteristics of the data flow, and perform real-time classification of data changes; design an adaptive sampling strategy based on the priority matrix, and implement fine-grained collection of priority data above a preset adaptive threshold;

[0140] The second unit is used to construct a multi-dimensional evaluation index using the node performance index, and the multi-dimensional evaluation index includes a node load index, a network congestion level, and a disk read and write pressure; the multi-dimensional evaluation index is used as an input of a time series prediction model, the historical change trend of the node performance index is analyzed, and a load prediction value in a future time window is generated; the available resource capacity of each node is calculated according to the load prediction value; the data change events in the real-time data synchronization queue are dynamically allocated based on the available resource capacity, and the data change events are encapsulated as data synchronization tasks; a load balancing matrix is ​​constructed based on the node load index, and the data synchronization tasks are allocated to nodes whose load values ​​are lower than a load threshold;

[0141] The third unit is used to start the distributed transaction coordinator to process the data synchronization tasks in parallel; use the two-phase commit protocol to process the data synchronization tasks to ensure data consistency; monitor the execution status of the data synchronization tasks in real time, record the transmission time of the data from the source to the target to obtain the synchronization delay time, compare the source data with the target data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage; when a synchronization anomaly is detected, the abnormal data synchronization task is put back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, the load rebalancing process is triggered based on the resource usage, and the load balancing matrix is ​​recalculated; the synchronization delay time, the consistency check result and the resource usage are used as feedback data to optimize the prediction accuracy of the time series prediction model and dynamically adjust the allocation strategy of the data synchronization task.

[0142] According to a third aspect of the embodiments of the present invention,

[0143] An electronic device is provided, comprising:

[0144] processor;

[0145] a memory for storing processor-executable instructions;

[0146] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0147] A fourth aspect of the embodiments of the present invention is:

[0148] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0149] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The CDC synchronization method based on Oracle RAC is characterized by: include: Deploy an incremental log parser at the database layer to capture atomic operations in Oracle redo logs, deploy a session state collector at the application layer to obtain concurrent transaction information, and deploy a resource monitor at the system layer to collect node performance indicators such as processor usage, memory occupancy, and input and output loads; input the atomic operations, the concurrent transaction information, and the node performance indicators into an intelligent classification model to identify the timing characteristics, load characteristics, and dependency characteristics of the data flow; construct a priority matrix based on the timing characteristics, load characteristics, and dependency characteristics of the data flow to perform real-time classification of data changes; design an adaptive sampling strategy based on the priority matrix to implement fine-grained collection of priority data above a preset adaptive threshold; The node performance index is used to construct a multi-dimensional evaluation index, which includes a node load index, a network congestion level, and a disk read and write pressure; the multi-dimensional evaluation index is used as an input of a time series prediction model, the historical change trend of the node performance index is analyzed, and a load prediction value in a future time window is generated; the available resource capacity of each node is calculated according to the load prediction value; the data change events in the real-time data synchronization queue are dynamically allocated based on the available resource capacity, and the data change events are encapsulated as data synchronization tasks; a load balancing matrix is ​​constructed based on the node load index, and the data synchronization tasks are allocated to nodes whose load values ​​are lower than the load threshold; Start the distributed transaction coordinator to process the data synchronization tasks in parallel; use the two-phase commit protocol to process the data synchronization tasks to ensure data consistency; monitor the execution status of the data synchronization tasks in real time, record the transmission time of the data from the source to the target to obtain the synchronization delay time, compare the source data with the target data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage; when a synchronization anomaly is detected, put the abnormal data synchronization task back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, trigger the load rebalancing process based on the resource usage, and recalculate the load balancing matrix; use the synchronization delay time, the consistency check result and the resource usage as feedback data to optimize the prediction accuracy of the time series prediction model and dynamically adjust the allocation strategy of the data synchronization task.

2. The method according to claim 1, characterized in that Inputting the atomic operation, the concurrent transaction information and the node performance index into an intelligent classification model to identify the timing characteristics, load characteristics and dependency characteristics of the data flow; Based on the timing characteristics, load characteristics, and dependency characteristics of the data flow, a priority matrix is ​​built to classify data changes in real time, including: Inputting the atomic operation, the concurrent transaction information and the node performance index into an intelligent classification model, wherein the intelligent classification model adopts a deep neural network structure; extracting data flow features through the intelligent classification model, wherein the data flow features include timing features, load features and dependency features; Calculate a feature vector based on the data stream feature; input the feature vector into an adaptive weight calculation unit to generate a feature weight coefficient; multiply the feature weight coefficient by the feature vector to construct a priority matrix; Based on the priority matrix, data changes are graded in real time, and data changes are divided into priority data higher than a preset adaptive threshold and priority data lower than the preset adaptive threshold; fine-grained collection is implemented for the priority data higher than the preset adaptive threshold, and the fine-grained collection includes: establishing a dedicated collection channel, capturing the complete change sequence in real time, and recording related transaction information.

3. The method according to claim 1, characterized in that Designing an adaptive sampling strategy according to the priority matrix and implementing fine-grained collection of priority data above a preset adaptive threshold includes: Mapping the priority information in the priority matrix to a preset interval by a maximization-minimization normalization method to obtain a normalized priority value; Based on the normalized priority value, an adaptive threshold calculation model is established in the sliding time window, and an exponential smoothing algorithm is used to calculate the adaptive threshold in combination with the mean and standard deviation of the priority value in the current time window, wherein the exponential smoothing algorithm includes a smoothing coefficient and a threshold adjustment coefficient; Comparing the normalized priority value with the adaptive threshold, and for priority data higher than the adaptive threshold, dynamically adjusting the sampling frequency based on the difference between the normalized priority value and the adaptive threshold, wherein the sampling frequency increases as the difference increases; Constructing a double-ended queue for priority mutation monitoring, setting a priority mutation detection threshold, and adjusting the length of the double-ended queue adaptively with changes in the priority mutation detection threshold; Monitor the changes of the normalized priority value in real time. When it is detected that the normalized priority value jumps relative to the priority mutation detection threshold, increase the number of sampling bits from the basic number of bits to a higher number of bits, triggering the high-frequency sampling mode to perform fine-grained collection of priority data higher than the preset adaptive threshold.

4. The method according to claim 1, characterized in that Using the node performance indicators to construct multi-dimensional evaluation indicators, the multi-dimensional evaluation indicators include node load index, network congestion level and disk read and write pressure; using the multi-dimensional evaluation indicators as input to the time series prediction model, analyzing the historical change trend of the node performance indicators, and generating load prediction values ​​in the future time window; Calculating the available resource capacity of each node according to the load prediction value includes: Constructing multi-dimensional evaluation indicators based on the node performance indicators, calculating the node load index, which is based on a weighted combination of processor utilization and memory pressure; calculating the network congestion level, which is comprehensively evaluated by network throughput, data packet delay and packet loss rate; calculating the disk read and write pressure, which is evaluated in combination with IO response time and storage queue depth; Organize the multi-dimensional evaluation indicators in time series to form an indicator sequence; perform data cleaning on the indicator sequence to eliminate noise points and outliers; extract the long-term change trend, periodic fluctuation characteristics and burst mode of the indicator sequence; build a time series prediction model, which adopts a deep neural network architecture; input the long-term change trend, the periodic fluctuation characteristics and the burst mode into the time series prediction model; analyze the historical change rules based on the time series prediction model to generate load prediction values ​​in the future time window; According to the load prediction value, the node resource status is evaluated, and the available computing power of the processor is calculated, and the available computing power takes into account the multi-core task scheduling efficiency; the available memory capacity is estimated, and the available memory capacity is based on the page replacement rate and the degree of memory fragmentation; the network bandwidth margin is calculated, and the network bandwidth margin is combined with the link utilization and the transmission queue status; the storage system capacity is evaluated, and the storage system capacity is determined by the IO throughput and cache efficiency; a resource competition model is established to analyze the degree of interference between different load types; and the available resource capacity is adjusted according to the degree of interference.

5. The method according to claim 1, characterized in that Dynamically allocating data change events in the real-time data synchronization queue based on the available resource capacity, and encapsulating the data change events into data synchronization tasks; Building a load balancing matrix based on the node load index and allocating the data synchronization task to the node whose load value is lower than the load threshold comprises: Dynamically allocate data change events in the data synchronization queue according to the available resource capacity, monitor the waiting time of the data change event, and when the waiting time exceeds the preset time, increase the event priority through an aging algorithm; identify continuous change events of the same data table in the data synchronization queue, and integrate the continuous change events into batch tasks; generate data synchronization tasks based on the available resource capacity, the event priority and the batch tasks; Construct a load balancing matrix, which includes a node resource status dimension, a synchronization task feature dimension, and a historical execution effect dimension; in the node resource status dimension, record the node's processor usage, memory occupancy, and input and output pressure, and calculate the node's current load value; in the synchronization task feature dimension, record the resource demand and execution priority of the data synchronization task; in the historical execution effect dimension, record the node's task completion rate and response delay; The data synchronization task is allocated based on the load balancing matrix, and the node with the lowest current load value and meeting the resource demand is selected as the target node; the data synchronization task is allocated to the target node; the load status of the target node is continuously monitored, and when the current load value of the target node exceeds the load threshold, the task migration mechanism is triggered; the load balancing matrix information of other nodes is re-acquired, and the node with the lowest load is selected as the migration target.

6. The method according to claim 5, characterized in that Start the distributed transaction coordinator to process the data synchronization task in parallel; use the two-phase commit protocol to process the data synchronization task to ensure data consistency; monitor the execution status of the data synchronization task in real time, record the transmission time of data from the source end to the target end to obtain the synchronization delay time, compare the source end data with the target end data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage, including: Starting a distributed transaction coordinator, analyzing the dependencies of the data synchronization tasks, and constructing a task dependency graph; identifying a task subset without dependencies based on the task dependency graph, and marking the task subset as a parallel task group; monitoring system resource status, and dynamically calculating the maximum concurrency number based on processor load and memory usage; and allocating tasks to the parallel task group based on the maximum concurrency number; Before executing the parallelizable task group, a two-phase commit protocol is started; in the first phase, a prepare request is sent to the participating nodes, the prepare request includes synchronization data content and data table structure information; the participating nodes obtain an exclusive lock on the data table and verify the consistency of the data table structure; the synchronization data is written into a temporary storage area and a data checksum is generated; the participating nodes return a prepare response, the prepare response includes data checksum and resource reservation status; in the second phase, a decision is made whether to commit based on the prepare responses of all participating nodes; Collect the execution status of the synchronization process in real time, record the transmission time of data from the source to the target to obtain the synchronization delay time; compare the source data checksum with the target data checksum to generate a consistency check result; collect the resource usage of the participating nodes, including processor usage, memory occupancy and network bandwidth occupancy; calculate performance indicators based on the execution status, and compare the performance indicators with preset performance thresholds; generate performance alarms when the performance indicators exceed the preset performance thresholds.

7. The method according to claim 6, characterized in that When a synchronization anomaly is detected, the abnormal data synchronization task is put back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, the load rebalancing process is triggered based on the resource usage, and the load balancing matrix is ​​recalculated; the synchronization delay time, the consistency check result and the resource usage are used as feedback data to optimize the prediction accuracy of the time series prediction model, and the allocation strategy of the data synchronization task is dynamically adjusted, including: When the consistency check result shows that the data is inconsistent, the current synchronization state is recorded to obtain a synchronization exception mark; the data synchronization task with the synchronization exception mark and the resource usage are packaged together and put back into the real-time data synchronization queue; the task priority is calculated according to the synchronization delay time, and the real-time data synchronization queue is prioritized; when the synchronization delay time exceeds the preset time threshold, the load balancing matrix is ​​recalculated based on the resource usage; the load balancing matrix is ​​used to quantify the computing node load level; For the nodes whose load levels exceed the preset load threshold, the resource usage is analyzed and a load rebalancing process is started; the target nodes whose load levels are lower than the preset load threshold are identified from the load balancing matrix and a task migration plan is calculated; the synchronization delay time, the consistency check result and the resource usage are integrated into a performance evaluation index; the performance evaluation index is used as feedback data and input into a timing prediction model; Utilize the feedback data to analyze the trend of system performance changes and update the parameters of the timing prediction model; predict the system load in the future time window based on the updated timing prediction model; recalculate the load balancing matrix according to the predicted system load; and dynamically adjust the allocation strategy of the data synchronization task based on the updated load balancing matrix.

8. A CDC synchronization system based on Oracle RAC, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to deploy an incremental log parser at the database layer to capture atomic operations in Oracle redo logs, deploy a session state collector at the application layer to obtain concurrent transaction information, and deploy a resource monitor at the system layer to collect node performance indicators such as processor usage, memory occupancy, and input and output loads; input the atomic operations, the concurrent transaction information, and the node performance indicators into an intelligent classification model to identify the timing characteristics, load characteristics, and dependency characteristics of the data flow; construct a priority matrix based on the timing characteristics, load characteristics, and dependency characteristics of the data flow to perform real-time classification of data changes; design an adaptive sampling strategy based on the priority matrix to implement fine-grained collection of priority data above a preset adaptive threshold; The second unit is used to construct a multi-dimensional evaluation index using the node performance index, and the multi-dimensional evaluation index includes a node load index, a network congestion level, and a disk read and write pressure; the multi-dimensional evaluation index is used as an input of a time series prediction model, the historical change trend of the node performance index is analyzed, and a load prediction value in a future time window is generated; the available resource capacity of each node is calculated according to the load prediction value; the data change events in the real-time data synchronization queue are dynamically allocated based on the available resource capacity, and the data change events are encapsulated as data synchronization tasks; a load balancing matrix is ​​constructed based on the node load index, and the data synchronization tasks are allocated to nodes whose load values ​​are lower than a load threshold; The third unit is used to start the distributed transaction coordinator to process the data synchronization tasks in parallel; use the two-phase commit protocol to process the data synchronization tasks to ensure data consistency; monitor the execution status of the data synchronization tasks in real time, record the transmission time of the data from the source to the target to obtain the synchronization delay time, compare the source data with the target data to obtain the consistency check result, and collect the node performance indicators to obtain the resource usage; when a synchronization anomaly is detected, the abnormal data synchronization task is put back into the real-time data synchronization queue; when the synchronization delay time exceeds the preset threshold, the load rebalancing process is triggered based on the resource usage, and the load balancing matrix is ​​recalculated; the synchronization delay time, the consistency check result and the resource usage are used as feedback data to optimize the prediction accuracy of the time series prediction model and dynamically adjust the allocation strategy of the data synchronization task.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images