A method and system for data synchronization from multiple data sources

Abstract

This invention discloses a data synchronization method and system for multiple data sources, relating to the field of data synchronization technology. The method includes the following steps: performing bandwidth evaluation and topology testing on the data transmission channels between the source and target ends to obtain channel feature data and calculate priority scores, thereby dividing the channel into primary and backup channels; inputting the channel feature data into a long short-term memory network model based on an improved white whale optimization algorithm to obtain feature prediction values ​​and calculate channel health; switching the primary channel to a backup channel when its health is below a threshold; acquiring data to be backfilled, constructing a backfilled directed acyclic graph and inputting it into a graph convolutional network model to output backfilling priority scores and complete data backfilling; performing consistency verification on the target end for the data to be backfilled, storing the data after successful verification; acquiring channel anomaly feature data and transmission parameters, inputting them into a deep deterministic policy gradient model based on entropy regularization to obtain optimal transmission parameters.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, specifically to a data synchronization method and system for multiple data sources. Background Technology

[0002] In the field of information technology, with the continuous expansion of enterprise data scale and the increasing demand for real-time business operations, achieving efficient and reliable data synchronization between multiple data sources has become crucial. This process highly depends on the stability of the data transmission channel between the source and destination ends. The performance characteristics of the channel, such as latency, packet loss rate, and throughput, are a direct reflection of the dynamically changing network environment. Accurately perceiving and predicting the changing trends of these channel characteristics is the core foundation for ensuring the quality and continuity of data synchronization services.

[0003] Traditional methods employ a static monitoring mechanism based on fixed thresholds to evaluate channel status. This involves continuously collecting operational data from the channels (such as round-trip latency and packet loss rate) and setting empirical thresholds for each key performance indicator. When the system detects that a channel's characteristic indicator (such as latency) consistently exceeds its preset threshold, it triggers an alarm or switching logic, migrating data traffic from the current channel to a backup channel.

[0004] However, traditional methods rely on lagging judgments of the channel's history and current state, lacking the ability to proactively predict future degradation trends in channel performance. As a result, they can only respond passively after the channel performance has actually deteriorated. This leads to an unavoidable delay between the occurrence of an anomaly and the completion of the switchover, ultimately resulting in increased data synchronization delays or even brief interruptions, making it difficult to meet the stringent requirements for seamless data synchronization and switching. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a data synchronization method and system for multiple data sources, thereby resolving the problems existing in the background technology.

[0006] To achieve the above objectives, the present invention provides a data synchronization method and system for multiple data sources, comprising the following steps:

[0007] Step S1: Perform bandwidth evaluation and topology testing on the data transmission channel between the source and target ends to obtain channel characteristic data; based on the channel characteristic data, calculate the channel priority score, and divide the data transmission channel into the main channel and the backup channel according to the channel priority score;

[0008] Step S2: Input the channel feature data into the long short-term memory network model based on the improved white whale optimization algorithm, and output the channel feature prediction value. Based on the channel feature prediction value, calculate the channel health. When the channel health of the main channel is less than the preset threshold, the data transmission is switched from the main channel to the backup channel.

[0009] Step S3: Obtain the data to be supplemented after converting the main channel to the backup channel, construct a directed acyclic graph for supplementation based on the data to be supplemented, and input it into the graph convolutional network model. Output the supplementation priority score, and supplement the data to be supplemented based on the supplementation priority score.

[0010] Step S4: After receiving the data to be replenished, the target end performs consistency verification on the data to be replenished. After the consistency verification is successful, the data to be replenished is stored in the target end.

[0011] Preferably, the step of performing bandwidth evaluation and topology testing on the data transmission channel between the source and target ends to obtain channel characteristic data includes the following specific steps:

[0012] Probe packets are continuously sent to the data transmission channel between the source and the target to obtain the channel feature vector of each data transmission channel. The channel feature vector includes: round-trip delay, packet loss rate and throughput, transmission queue backlog depth and retry event density.

[0013] Preferably, the calculation of channel priority score based on channel feature data includes the following steps:

[0014] Based on channel feature data, latency score, packet loss rate score, and throughput score are calculated, thereby calculating channel priority score:

[0015] ;

[0016] in, Score the channel priority. To delay scoring, To score the packet loss rate, For throughput score, For delayed score weights, As the weight for packet loss rate score, Weights are assigned to throughput scores.

[0017] Preferably, the step of inputting channel feature data into a long short-term memory network model based on an improved beluga optimization algorithm and outputting predicted channel feature values ​​includes the following steps:

[0018] Population initialization based on reverse learning: Half of the original random solutions X are randomly generated, covering the hyperparameter search range of the Long Short-Term Memory (LSTM) network; subsequently, the reverse solution X' is calculated for each original random solution X, and its calculation formula is as follows:

[0019] X'=k ( + )-X;

[0020] Where k is the scaling factor. and Let be the lower and upper bounds of the cc-th hyperparameter;

[0021] The original random solutions and reverse solutions are merged, and the solution with the best fitness is selected through a fitness function to form the final initial population. The attraction mechanism of the firefly algorithm is introduced: during the exploration phase of the algorithm, for any two solutions in the population, the magnitude of the attraction β is related to the fitness difference and Euclidean distance between the individuals, as shown in the formula:

[0022] β= ;

[0023] Where β is the attractive force, The maximum attractive force is 1 by default; gx is the light absorption coefficient, which is 0.01 by default; and r is the distance between individuals.

[0024] Solutions with poor fitness will move towards solutions with better fitness. The magnitude of this movement is determined by both the attraction factor β and the random perturbation term, as shown in the formula:

[0025] ;

[0026] in, This represents the position of the m-th individual in the population during the dd-th iteration. This represents the updated position of the m-th individual in the (dd+1)-th iteration. This indicates that in the dd-th iteration, the fitness of the population is better than... The position of the m2th individual, bc is the step size factor which defaults to 0.05, and rand(0,1) is a random number between 0 and 1;

[0027] By calculating the variance of population fitness, if the variance is less than a preset threshold, the population is deemed to have insufficient diversity. Then, for the current optimal solution... Generating nonlinear inverse solutions based on lens imaging principles The calculation formula is: '= + - This nonlinear mapping ensures that the inverse solution is both far from the current local optimum and within a reasonable range, effectively expanding the search space.

[0028] Population update: The generated reverse solution Replace the worst-fitting solution in the population, recalculate the population fitness, break the local optimum constraint, and increase the probability of finding the global optimal hyperparameter combination; after the population update is completed, the algorithm continues to enter the next iteration, repeating the optimization process of the exploration and development stage until the preset maximum number of iterations is reached, and finally the hyperparameter-optimized long short-term memory network model is obtained.

[0029] Channel feature data TZ(t) within time window t = [ [D(t), Q(t), R(t), J(t)], including the packet loss rate within time window t. The round-trip delay D(t) under time window t, the send queue backlog depth Q(t) under time window t, the retry event density R(t) under time window t, and the throughput J(t) under time window t;

[0030] Input the channel feature data [TZ(1),TZ(2),TZ(3),...,TZ(t)] of t time windows into a trained long short-term memory network model based on the improved white whale optimization algorithm, and output the predicted channel feature values ​​for the next t+1 time windows. , =[ , (t+1) (t+1) (t+1) [(t+1)], where, This is the predicted packet loss rate for time window t+1. (t+1) represents the predicted round-trip delay within the time window t+1. (t+1) represents the predicted value of the sending queue backlog depth under time window t+1. (t+1) represents the predicted retry event density within time window t+1. (t+1) represents the predicted throughput value within the time window t+1.

[0031] Preferably, the channel health status is:

[0032] ;

[0033] in, Here, t represents the channel health score, T represents the time window index, and c represents the time window length threshold. This represents the total number of features in the channel feature data. This represents the predicted value of the c-th feature in the channel feature data within the time window t+1. This represents the true value of the c-th feature in the channel feature data within the time window t+1. To prevent division by zero constant, the default value is 0. .

[0034] Preferably, the construction of the directed acyclic graph based on the data to be replenished includes the following steps:

[0035] Retrieve the data to be replenished generated when the primary channel is converted to the backup channel; parse the reference dependencies between data units based on the transaction log, including strong dependencies and weak dependencies;

[0036] Based on the above reference dependencies, a Directed Acyclic Graph (DAG) is constructed for data restoration: each node in the DAG represents a data unit to be restored, and the node feature vector includes data size, business criticality level, timestamp, and channel source; each edge represents a reference dependency relationship, with the direction pointing from the referenced data unit to the referencing data unit; the edge weight is quantized and assigned based on the dependency type: high weight 0.9 corresponds to strong dependency; medium weight 0.6 corresponds to weak dependency; low weight 0.1 corresponds to data units without direct reference but with time sequence relationship.

[0037] Preferably, the data replenishment based on the replenishment priority score includes the following specific steps:

[0038] Based on the completion priority score output by the graph convolutional network model, dependency reordering priority revision is performed, and the revision is based on the dependency reordering priority coefficient. Combining the dependency type of each data unit in the backfilled directed acyclic graph, the dependent nodes in the strong dependency relationship with an edge weight of 0.9 are multiplied by the original backfilling priority score output by the graph convolutional network model. For weakly dependent nodes with an edge weight of 0.6 and temporally related nodes with an edge weight of 0.1, the backfill priority score remains unchanged; finally, the revised backfill priority score is generated.

[0039] Based on the revised backfill priority score, a backfill plan is generated and executed: data units are sorted in descending order of backfill priority score, and data backfill is performed in sequence.

[0040] Preferably, after receiving the data to be replenished, the target end performs consistency verification on the data to be replenished, including the following specific steps:

[0041] After receiving the data to be replenished, the target end aggregates all channel data units according to the primary key of the data units in the data to be replenished. The aggregation condition formula is as follows: ={ | =PK and All channels have data to be written, where... This represents an aggregated data set with a primary key (PK). For a single data unit, The primary key value of the data unit;

[0042] First, check if any data units in the aggregate collection have the same version number and identical field values. If such data units exist, they are considered duplicate data transmissions. In this case, the data unit with the latest timestamp is used as the unique valid unit for that version, and duplicates are removed. Next, check if there are different modifications to the same field within the same version number, using the following formula:

[0043] ;

[0044] in, For the field conflict determination results, Representing data unit The specific field names, , Representing data unit In the field value of Field, Representing data unit The specific field names, Representing data unit The field value in the Field field;

[0045] when There are conflicting fields with the same version. When the score is equal to 1, calculate the adjudication score for each data unit using the following formula:

[0046] ;

[0047] in, For data units The score of the ruling, As a weight for the credibility of the source, As a business criticality weight, Indicates the arrival timestamp of the data unit. This represents the maximum value of the arrival timestamps within the aggregate set. , , For strategy weights, + + =1;

[0048] choose The highest-valued data unit is designated as the sole valid unit for that version, and all other data units in that version are removed.

[0049] After processing aggregate sets with identical version conflicts, perform version number continuity verification using the following formula:

[0050] ;

[0051] in, For version verification results, For data units The version number, k represents the numerical value corresponding to the version number.

[0052] Preferably, step S5 is also included:

[0053] After the supplementary data is stored in the target end, the channel anomaly feature data and channel transmission parameters during the transmission process between the source end and the target end are obtained and input into the deep deterministic policy gradient model based on entropy regularization. The optimal channel transmission parameters are output and the data transmission channel is optimized based on the optimal channel parameters.

[0054] A data synchronization system with multiple data sources includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0055] This invention provides a data synchronization method for multiple data sources, involving machine learning and deep learning technologies, which has the following beneficial effects:

[0056] (1) The Long Short-Term Memory (LSTM) network model can effectively process time series data and capture the dynamic changes of channel characteristics (such as packet loss rate and latency). By predicting the channel characteristics of future time windows, the channel health can be calculated in advance, and the active switching from the main channel to the backup channel can be realized, thereby improving the real-time performance and reliability of data synchronization and avoiding data loss or interruption due to channel abnormalities.

[0057] (2) The improved white whale optimization algorithm enhances global search and local development capabilities by introducing optimization mechanisms such as reverse learning and firefly algorithm strategies, thus avoiding getting trapped in local optima. When optimizing the hyperparameters of the long short-term memory network model (such as the number of hidden layer neurons and learning rate), the improved white whale optimization algorithm can improve the model's prediction accuracy and generalization performance, ensure the accuracy of channel health assessment, and provide a reliable basis for system switching decisions.

[0058] (3) The deep deterministic strategy gradient model can adaptively adjust the channel transmission parameters (such as switching threshold and shadow synchronization sampling rate) according to the real-time status of the channel (such as abnormal features and performance indicators). By continuously learning the optimal strategy, the model can dynamically optimize the data transmission process, improve switching efficiency, reduce latency, and enhance the system's adaptability to complex network environments, thereby improving the overall synchronization performance.

[0059] (4) Entropy regularization is adopted to introduce policy entropy into the deep deterministic policy gradient model, which encourages policy exploration and prevents premature convergence to suboptimal solutions. By balancing exploration and utilization, entropy regularization ensures that the model maintains diversity during training, can discover more robust and adaptive policies, improves stability and overall performance in uncertain environments, and reduces the risks caused by parameter fixation. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating the steps of a multi-data source data synchronization method proposed in this invention.

[0062] Figure 2 This is a hierarchical diagram of the steps involved in converting a primary channel to a backup channel in a multi-data source data synchronization method proposed in this invention.

[0063] Figure 3 This is a hierarchical diagram of the steps involved in receiving data to be replenished at the target end in a multi-data source data synchronization method proposed in this invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Please see Figures 1-3 The present invention provides a technical solution: a data synchronization method for multiple data sources.

[0066] Step S1: Perform bandwidth evaluation and topology testing on the data transmission channel between the source and target ends to obtain channel characteristic data; based on the channel characteristic data, calculate the channel priority score, and divide the data transmission channel into the main channel and the backup channel according to the channel priority score.

[0067] When establishing data transmission between the source and destination, at least two independent synchronization channels must be constructed. These channels should meet physical or logical isolation requirements, such as isolating fault domains through fiber optic links from different network service providers or transmission paths using different network protocols. The channel architecture must clearly define roles: one channel should be designated as the primary channel, responsible for real-time data synchronization; the remaining channels should serve as backup channels, in standby mode. Only one primary channel should exist at any given time, and the number of backup channels can be expanded according to reliability requirements.

[0068] It should be noted that the source end is the origin of data generation and storage, acting as the initiator of the synchronization process; the target end is the receiver of data, receiving data from different source ends.

[0069] To achieve fault isolation and precise switching between channels, a dedicated logical resource system needs to be established for each channel. Specifically, each channel needs to independently maintain the following three core components: 1. Transmit queue: A first-in, first-out (FIFO) data structure is used to cache the data units to be transmitted in this channel, ensuring data order; 2. Receive acknowledgment set: An efficient query data structure (such as a hash table or index table) independently maintained by each synchronization channel, specifically recording the unique identifier of the data unit that has been successfully persisted at the target end after being transmitted through this channel, serving as the sole proof of data transmission completion; 3. Retry logbook: Stores data of channel transmission failures and retry status.

[0070] After channel establishment is completed, topology probing and bandwidth evaluation are performed on all channels: First, probe packets of a specific size are continuously sent to each channel to obtain channel characteristic data, including: round-trip latency, packet loss rate, throughput, send queue backlog depth, and retry event density. Based on the channel characteristic data, latency score, packet loss rate score, and throughput score are calculated.

[0071] It should be noted that the delay score of the channel feature data for each channel is... The packet loss rate score in the channel feature data for each channel is: The throughput score in the channel feature data for each channel is: If the maximum round-trip delay value of all channels is equal to the minimum round-trip delay value (denominator is 0), the delay score of that channel is set to 1 by default; if the maximum packet loss rate of all channels is equal to the minimum packet loss rate (denominator is 0), the packet loss rate score of that channel is set to 1 by default; if the maximum throughput of all channels is equal to the minimum throughput (denominator is 0), the throughput score of that channel is set to 1 by default.

[0072] Calculate channel priority score based on channel feature data:

[0073] ;

[0074] in, Score the channel priority. To delay scoring, To score the packet loss rate, For throughput score, For delayed score weights, As the weight for packet loss rate score, Weights are assigned to throughput scores.

[0075] It should be noted that the aforementioned delay score weight Packet loss rate score weight Throughput score weighting Based on the needs of the business scenario, this can be quantified through a business requirement level-weight mapping: First, define the business's requirement level for each indicator (high / medium / low), referencing industry SLA (Service Level Agreement) indicators (e.g., for financial transaction systems, real-time requirement level is high, packet loss rate requirement level is high; for data backup scenarios, integrity requirement level is high, real-time requirement level is medium); allocate weight ranges according to the requirement level (high: 0.4-0.6, medium: 0.2-0.4, low: 0.1-0.2), and the sum of the three weights is 1. For example, for financial transaction systems with extremely high real-time requirements, latency score weights should be assigned. A larger value can be taken It is 0.5. and Both are 0.25; however, for data backup scenarios that are sensitive to integrity, the weighting of the packet loss rate score needs to be increased. Then it is acceptable. It is 0.5. and Both are 0.25.

[0076] It should be noted that when calculating channel priority scores, only latency scores, packet loss rate scores, and throughput scores are focused on. The core reason is that the main objective at this stage is to complete the initial allocation of primary and backup channels. It is necessary to prioritize the most basic and fundamental transmission capabilities of the channels. Latency directly affects the real-time performance of data synchronization, packet loss rate affects the integrity of data transmission, and throughput determines the efficiency of the channel in carrying data. These three indicators together constitute the core foundation for whether the channel can transmit data stably and efficiently. Moreover, they can be directly obtained by sending probe data packets during the channel initialization and probing phase. The sending queue compression depth and retry event density are dynamic status indicators generated during channel operation and data transmission. They have not yet been generated during the initialization and primary / backup selection phase and do not need to be included in the calculation. At the same time, using only these three core indicators can greatly simplify the priority calculation logic and avoid the computational redundancy caused by multi-dimensional indicators.

[0077] After calculating the channel priority score, all data transmission channels at the target end and one source end are sorted according to their channel priority scores. The channel with the highest channel priority score is designated as the main channel for the target end and the source end, responsible for carrying the real-time data stream. All other channels are listed as backup channels in descending order of priority score, forming a clear backup priority queue.

[0078] It should be noted that a target end (data receiver) typically establishes synchronization relationships with multiple different source ends (data senders) to aggregate data from various sources. For each pair of source and target ends, multiple data transmission channels are pre-established, and then the primary channel and backup channels are designated.

[0079] Step S2: Input the channel feature data into the long short-term memory network model based on the improved white whale optimization algorithm, and output the channel feature prediction value. Based on the channel feature prediction value, calculate the channel health. When the channel health of the main channel is less than the preset threshold, the data transmission is switched from the main channel to the backup channel.

[0080] For each time window t (e.g., 60 seconds), the channel feature data includes the packet loss rate. Calculate within a fixed time window t ,in This represents the number of data packets sent within the window. The number of corresponding acknowledgment packets. Round-trip delay D(t): Records the round-trip delay from sending to receiving an acknowledgment for each data packet, and calculates the average round-trip delay of all data packets as the round-trip delay D(t) for this time window. Send queue backlog depth Q(t): Directly monitors the instantaneous length of the send queue (e.g., the number of messages to be sent). Retry event density R(t): The number of retry events that occur within window t (monitored by filling the record book to detect the number of retries within window t). Throughput J(t): The total number of data successfully transmitted within window t.

[0081] Channel feature data TZ(t) within time window t = [ [D(t), Q(t), R(t), J(t)], including the packet loss rate within time window t. The round-trip delay D(t) under time window t, the send queue backlog depth Q(t) under time window t, the retry event density R(t) under time window t, and the throughput J(t) under time window t.

[0082] The channel feature data [TZ(1),TZ(2),TZ(3),...,TZ(t)] of t time windows are input into the long short-term memory network model based on the improved beluga optimization algorithm.

[0083] The improved beluga whale optimization algorithm employs a population initialization strategy based on back-learning in the initialization phase. This involves calculating the symmetric solution for each randomly generated hyperparameter combination and selecting the best solutions from the merged original and symmetric solutions to form the initial population, thus improving the quality of the starting point. Secondly, in the exploration phase (simulating a broad search of beluga whale swimming), a firefly algorithm strategy is introduced, attracting solutions with lower fitness to move towards solutions with higher fitness, enhancing the directionality and efficiency of global exploration. Finally, in the development phase (simulating fine-grained optimization of beluga whale predation), a lens-based back-learning strategy is combined. When population diversity decreases, a nonlinear back-learning solution for the current optimal solution is dynamically generated to effectively escape local optima.

[0084] Population initialization based on reverse learning: Half of the initial hyperparameter combinations (i.e., original random solutions X) are randomly generated, covering the hyperparameter search range of LSTM networks (20-100 hidden layer neurons, learning rate 0.001-0.05, number of iterations 50-200, dropout probability 0.1-0.3); subsequently, for each original random solution X, its reverse solution X' is calculated, with the formula X'=k ( + )-X, where k is the scaling factor, and Let be the lower and upper bounds of the cc-th hyperparameter. The original random solutions and reverse solutions are merged, and the solution with the best fitness is selected through the fitness function (i.e., the mean square error of the prediction of the LSTM model) to form the final initial population, making the initial population more evenly distributed in the solution space. The traditional beluga optimization algorithm uses pure random search in the exploration phase (simulating beluga swimming extensive search), which suffers from low exploration efficiency and difficulty in clustering towards high-quality solution regions. The firefly algorithm's attraction mechanism is introduced: in the exploration phase of the algorithm (simulating beluga swimming extensive search), the firefly algorithm's attraction mechanism is introduced to enhance the guidance of the exploration. For any two solutions in the population, individuals with poorer fitness will be attracted to individuals with better fitness and move towards them. The magnitude of the attraction β is related to the fitness difference and Euclidean distance between individuals, specifically β = ,in The maximum attractive force is 1 by default, gx is the light absorption coefficient (default 0.01), and r is the distance between individuals. Position update: Solutions with poor fitness will move towards solutions with better fitness. The magnitude of the movement is determined by the attractive force β and the random perturbation term, as shown in the formula:

[0085] ;

[0086] in, This represents the position of the m-th individual in the population during the dd-th iteration. This represents the updated position of the m-th individual in the (dd+1)-th iteration. This indicates that in the dd-th iteration, the fitness of the population is better than... The position of the m2th individual, bc is the step size factor which is 0.05 by default, and rand(0,1) is a random number between 0 and 1.

[0087] Traditional beluga whale optimization algorithms are prone to getting stuck in local optima during the development phase (simulating fine-grained optimization of beluga whale predation) due to declining population diversity. A dynamic back-learning strategy based on the lens imaging principle is used for optimization: by calculating the variance of the population fitness, when the variance is less than a preset threshold (default 0.001), it is determined that the population diversity is insufficient and may be stuck in a local optimum, and this strategy is immediately activated; for the current optimal solution... Generating nonlinear inverse solutions based on lens imaging principles The calculation formula is: '= + - This nonlinear mapping ensures that the inverse solution is both far from the current local optimum and within a reasonable range, effectively expanding the search space.

[0088] Population update: The generated reverse solution Replace the worst-fitting solution in the population, recalculate the population fitness, break the local optimum constraint, and increase the probability of finding the globally optimal hyperparameter combination. After the population update is completed, the algorithm continues to the next iteration, repeating the optimization process of the exploration and development phases until the preset maximum number of iterations (e.g., 100 times) is reached or the population fitness converges to a stable range (the standard deviation of the fitness function is less than 0.0005 for 5 consecutive iterations). Finally, the hyperparameter-optimized Long Short-Term Memory network model is obtained.

[0089] Historical channel feature data were collected and divided into training and test sets in a 7:3 ratio. Model training and prediction were then performed. Training process: The Adam optimizer was used with MSE as the loss function. The training set was input into the long short-term memory network model optimized by the improved white whale optimization algorithm for training. Overfitting was prevented through an early stopping mechanism.

[0090] Input the channel feature data [TZ(1),TZ(2),TZ(3),...,TZ(t)] of t time windows into a trained long short-term memory network model based on the improved white whale optimization algorithm, and output the predicted channel feature values ​​for the next t+1 time windows. , =[ , (t+1) (t+1) (t+1) [(t+1)], where, This is the predicted packet loss rate for time window t+1. (t+1) represents the predicted round-trip delay within the time window t+1. (t+1) represents the predicted value of the sending queue backlog depth under time window t+1. (t+1) represents the predicted retry event density within time window t+1. (t+1) represents the predicted throughput value within the time window t+1.

[0091] Real-time acquisition of channel feature vector values, calculation of the channel feature vector values ​​and predicted channel feature values ​​to obtain channel health:

[0092] ;

[0093] in, Here, t represents the channel health score, T represents the time window index, and c represents the time window length threshold. This represents the total number of features in the channel feature data. This represents the predicted value of the c-th feature in the channel feature data within the time window t+1. This represents the true value of the c-th feature in the channel feature data within the time window t+1. To prevent division by zero constant, the default value is 0. .

[0094] When the health of the main channel remains below a preset threshold of 0.6 for T consecutive time windows (e.g., T=2), a seamless switching process is immediately initiated: a freeze command is sent to the main channel's data transmitter to suspend the extraction of new data units from the transmission queue; the transmission queue and reception acknowledgment set of the main channel are scanned synchronously, and the position of the last data unit successfully acknowledged by the target end is recorded as the freeze boundary point. All transmitted but unacknowledged data units after this freeze boundary point constitute the unacknowledged interval, and a globally unique snapshot version number is generated for the unacknowledged interval as a marker (serving as the core index for data replenishment in step S3). Based on the backup priority queue generated in step S1, the backup channel with the highest priority is selected to take over the data transmission task and is promoted to the current main channel; the transmission tasks of the unacknowledged interval (associated with the snapshot version number) are transferred to the current main channel to ensure uninterrupted data synchronization.

[0095] After the channel switch is completed, the original main channel's operating mode is switched to keep-alive diagnostic mode: lightweight probe data packets (such as ICMP messages) are sent at a frequency of 30 seconds / time to collect packet loss rate, round-trip delay, sending queue backlog depth, retry event density, and throughput as real values. The packet loss rate, round-trip delay, sending queue backlog depth, retry event density, and throughput are predicted (same as the long-term network model in step S2). The predicted values ​​are compared with the real values ​​to calculate the channel health. After confirming that the channel performance indicators have recovered to a stable threshold (channel health ≥ 0.8), it is not immediately allowed to resume carrying service traffic. Instead, it is switched to the shadow synchronization strategy, that is, a small amount of sampled data from the current main channel is mirrored with very low priority and transmitted for verification. Throughout this process, all status information of the original main channel, such as acknowledgment reception records and retry records, is strictly updated in its independent logbook to ensure that its status repair and verification behavior will not cause any data or status-level interference or pollution to the current main channel.

[0096] It should be noted that the shadow synchronization strategy is a continuously enabled verification mechanism. This strategy applies to all standby channels in a standby state. During the normal operation of the standby channel, it uses front-end traffic mirroring technology to capture data segments flowing through the current primary channel (e.g., a 1% sampling rate), copies them, and injects them into the independent transmission path of the standby channel for verification, thereby continuously monitoring its link robustness. When the performance indicators of the original primary channel recover to a stable threshold, the shadow synchronization strategy is also used to verify the original primary channel. All status updates generated during the above verification process (such as acceptance records and retry records) are written to the independent logbook of each channel (standby channel and original primary channel) to ensure isolation from the current primary channel state and avoid interference. By fully executing the decoding, target-end confirmation response, and persistent disk write-in verification, the data processing capabilities and link robustness of the standby channel and the original primary channel are tested in real time. Once the original primary channel completes the verification of the specified period (e.g., 3 consecutive time windows) through the shadow synchronization strategy, and the channel health calculated during the verification period is stably maintained at a higher threshold (e.g., ≥0.9), it is determined that the channel has been fully recovered. Based on its performance during the verification period (e.g., round-trip delay, packet loss rate, and other channel characteristic data), its channel priority score is updated and it is added to the backup channel priority queue.

[0097] Step S3: Obtain the data to be supplemented after converting the main channel to the backup channel, construct a directed acyclic graph for supplementation based on the data to be supplemented, and input it into the graph convolutional network model. Output the supplementation priority score, and supplement the data to be supplemented based on the supplementation priority score.

[0098] The data to be replenished generated by converting the main channel to the backup channel consists of all data units in the unconfirmed interval after freezing nodes in the main channel in step S2. Each data unit contains: a unique identifier, such as a transaction ID, which serves as a globally unique index for the data unit and is used to associate the data unit with nodes in the directed acyclic graph (DAG) for replenishment (i.e., the unique identifier of each data unit corresponds to the "node ID" of a node in the DAG, ensuring a one-to-one mapping between data units and nodes); content load: the actual business data of the data unit (such as order details, inventory values), and its data size (such as the number of bytes) serves as one of the core dimensions of the DAG node feature vector; metadata: including timestamp (data generation / sending time) and source channel information (original data transmission channel identifier), which directly serve as the "timestamp" and "channel source" dimensions in the DAG node feature vector;

[0099] Based on transaction logs (such as database binlog), the reference dependencies between data units are parsed. Reference dependencies include strong dependencies (such as foreign key constraints and transaction order constraints, e.g., inventory update transactions must be committed before order generation transactions) and weak dependencies (such as logical references and business rule associations, e.g., user query operations depend on previous data insertions).

[0100] Based on the aforementioned reference dependencies, a Directed Acyclic Graph (DAG) is constructed for data replenishment. Each node in the DAG represents a data unit to be replenished, and the node feature vector includes data size, business criticality level, timestamp, and channel source. Each edge represents a reference dependency, with the direction pointing from the referenced data unit to the referencing data unit (e.g., if unit A is referenced by unit B, then the edge is A→B). Edge weights are assigned based on dependency type: high weight (0.9) corresponds to strong dependencies (such as foreign key constraints, transaction order constraints); medium weight (0.6) corresponds to weak dependencies (such as logical references, business associations); low weight (0.1) corresponds to data units without direct references but with time-series relationships (such as data generated in time order but without explicit dependencies). Weights are predefined through historical data statistics or business rules (e.g., in the order system, the dependency weight between inventory updates and order generation is set to 0.9, and the dependency weight between log recording and business operations is set to 0.1).

[0101] It should be noted that the "business criticality level" in the node feature vector is determined based on the business attributes of the data unit (extracted from the business tags of the content load, such as order payment data being marked as "high level" and ordinary log data being marked as "low level") or pre-configured business rules (such as core transaction data being set to high level by default and non-core query data being set to medium and low level), so as to achieve a complete correspondence between data unit information and node feature vector.

[0102] Convert the backfilled DAG into model input data: an adjacency matrix (storing edge weights, a sparse matrix with elements...) The adjacency matrix represents the edge weight from node o1 to node o2 (0 when there is no connection), and the node feature matrix (stores node features, with dimensions N×D, where N is the number of data unit nodes and D is the number of node features, D=4 here). The adjacency matrix and node feature matrix are input into the graph convolutional network model to calculate the backfill priority score for each data unit. The GCN model structure includes an input layer, two graph convolutional layers, and an output layer: the input layer receives the node feature matrix (dimension N×4); the first graph convolutional layer uses 32 neurons with the ReLU activation function; the second graph convolutional layer uses 16 neurons with the ReLU activation function; the output layer is a fully connected layer with the Sigmoid activation function, outputting the backfill priority score (range [0,1], higher scores indicate higher priority, e.g., units with scores >0.8 need to be processed immediately). The model aggregates neighbor node information through graph convolution operations; the core formula is:

[0103] ;

[0104] in, This represents the feature matrix of the nodes in the (l+1)th layer of the graph convolutional neural network. It is an adjacency matrix with self-connections. For degree matrix, Let L be the feature matrix of the l-th layer of the graph convolutional neural network. This is the weight matrix. () is the activation function.

[0105] The core formula updates the node feature matrix iteratively through multi-layer graph convolution to obtain the final layer feature matrix. The output layer of the subsequent graph convolutional neural network model will... The backfill priority score is mapped to [0,1] using a fully connected network and a Sigmoid activation function.

[0106] It should be noted that the training of the graph neural network model is based on the Directed Acyclic Graph (DAG) formed by dependency aggregation in the historical data recovery scenario. Each node in the DAG (corresponding to a historical data unit to be recovered) is used as a labeling object. First, according to business rules (such as high, medium, and low initial priorities for core transaction data with high business criticality, dependent nodes in strong dependencies, and nodes with small data volume and fast recovery speed, respectively corresponding to high, medium, and low initial priorities), an initial recovery priority score (label value) within the range of 0-1 is assigned to each node. Then, reinforcement learning is used to optimize the initial labeling, combined with the actual effect of historical recovery (such as increasing the label value of nodes that cause subsequent dependent nodes to be blocked due to low labeling priority, and decreasing the label value of nodes that have high labeling priority but do not help the overall process). The DAG (Data Amplification Group) aggregated from historical dependencies (containing a node feature matrix and an adjacency matrix; the node feature matrix records the data size, business criticality level, timestamp, and channel source of each historical node; the adjacency matrix records the edge weights of the dependencies between nodes) is used as input data. The optimized backfill priority label is used as the target output. The mean squared error is used as the loss function, and the Adam optimizer (learning rate 0.001) is used to train the graph convolutional network model. By iteratively adjusting the weight matrices of each layer of the model, the model gradually learns to integrate the node's own features (node ​​feature matrix) and the dependencies between nodes (edge ​​weights of the adjacency matrix), ensuring that the trained model can output priority judgment results that meet business requirements in actual backfill scenarios.

[0107] Based on the completion priority score output by the graph convolutional network model, dependency reordering priority revision is performed, and the revision is based on the dependency reordering priority coefficient. Based on the dependency type of each data unit in the repaired directed acyclic graph, for nodes in strong dependencies with an edge weight of 0.9, the original repair priority score output by the GCN is multiplied by [missing information]. For weakly dependent nodes with an edge weight of 0.6 and temporally related nodes with an edge weight of 0.1, the backfill priority score remains unchanged. Finally, the revised backfill priority score is generated.

[0108] It should be noted that the dependency reordering priority revision coefficient... ( >1 (initialized to 1.2), the dependency rearrangement priority revision coefficient is a key adjustment factor for the original imputation priority score and the actual imputation execution order output by the convolutional network of the connection graph. Based on the original imputation priority score, a secondary priority calibration is performed in combination with the dependency relationship type between the data units to be imputed. For strong dependencies with an edge weight of 0.9 in the directed acyclic graph being imputed, this coefficient will be applied to the dependent node in the strong dependency link by multiplying its original imputation priority score by 0.9. This method significantly improves priority ranking, ensuring that key units for subsequent node re-upload are processed first, avoiding blockage of the entire re-upload link due to delayed re-upload of strongly dependent nodes; while for weakly dependent nodes with an edge weight of 0.6 and time-series related nodes with an edge weight of 0.1, this... By not intervening and keeping the original replenishment priority score unchanged, we can avoid the impact of excessive adjustment on replenishment efficiency and ensure the rationality of the data unit replenishment order in scenarios with weak dependencies and temporal correlations. Ultimately, this differentiated adjustment ensures the correctness and efficiency of the replenishment process.

[0109] Based on the revised replacement priority score, a replacement plan is generated and executed: data units are sorted in descending order of score, high-scoring units are replaced first, and data units are replaced in order of sorting results.

[0110] Step S4: After receiving the data to be replenished, the target end performs consistency verification on the data to be replenished. After the consistency verification is successful, the data to be replenished is stored in the target end.

[0111] Before writing data to the target end, initiate a consistency verification process: capture all data units to be written in real time (including the data to be replenished in step S3 and other real-time data transmitted from the source end to the target end in the main channel), aggregate them according to the primary key of the data unit (such as "Order ID" in the order table and "User ID" in the user table), and generate a verification list entry; this entry aggregates all operation requests from different source ends (including the data to be replenished transmitted in step S3 and real-time data from other source ends in the main channel), and records the core information of each request, such as data version and arrival timestamp.

[0112] Each request also includes source credibility and business criticality level. Source credibility is determined by a preset weight based on the data source level (e.g., data sources with high reliability requirements such as core finance and government affairs have a weight of 0.9, data sources for ordinary business scenarios such as enterprise business and user management have a weight of 0.3, and non-core data sources such as logs and temporary caches have a weight of 0.1). Business criticality level weight is consistent with the "business criticality level" in the feature vector of step S3: core transaction data (e.g., order payment) is set to 1, ordinary business data (e.g., user query records) is set to 0.5, and non-core data (e.g., system logs) is set to 0.1.

[0113] Aggregate all channel data units based on their primary key. The aggregation formula is as follows: ,in, This represents an aggregated data set with a primary key (PK). For a single data unit, This is the primary key value of the data unit.

[0114] Check if any data units in the aggregate collection have the same version number and identical field values. If so, they are considered duplicate data transfers, and the data unit with the latest timestamp is used as the unique valid unit for that version, eliminating duplicates. Next, check if there are different modifications to the same field within the same version number, using the following formula:

[0115] ;

[0116] in, For the field conflict determination results, Representing data unit The specific field names, , Representing data unit In the field value of Field, Representing data unit The specific field names, Representing data unit The field value in the Field field.

[0117] when There is a conflict between fields of the same version (i.e.) When the score equals 1), calculate the adjudication score for each data unit using the following formula:

[0118] ;

[0119] in, For data units The score of the ruling, As a weight for the credibility of the source, As a business criticality weight, Indicates the arrival timestamp of the data unit. This represents the maximum value of the arrival timestamps within the aggregate set. , , For strategy weights, + + =1.

[0120] It should be noted that the strategy weights , , Based on specific scenarios, the allocation is tailored. For example, in financial systems, priority is given to business criticality (β=0.5) and source credibility (α=0.25), while also considering timeliness (γ=0.25). In backup scenarios, the integrity weight is increased (β=0.5) and other parameters are adjusted. 0.25 is acceptable. 0.25 is acceptable.

[0121] choose The highest-scoring data unit is designated as the sole valid unit for that version, and all other data units in that version are removed. If multiple highest-scoring data units have the same value, manual arbitration is triggered.

[0122] It's important to note that selecting the data unit with the highest adjudication score as the sole valid unit for that version and deleting other units is crucial. This score is calculated using a weighted average of source credibility (ensuring data comes from reliable data sources, such as data from core business systems being more reliable than data from peripheral systems), business criticality (prioritizing the validity of core business data, such as transaction order data being more critical than ordinary log data), and timestamp proportion (filtering the most recently arrived valid data to avoid using lagging data). Essentially, it comprehensively evaluates conflicting data units from three dimensions: data quality, business value, and timeliness. The data unit with the highest score minimizes the impact of erroneous data on the target business. Deleting other data units in that version is to completely eliminate data conflicts within the version (such as two different inventory values ​​for the same order ID), avoiding redundant storage on the target side or data ambiguity during subsequent queries / calculations. This ensures that each data version on the target side retains only one unique and accurate set of information, aligning with the core objective of "unique version, consistent data" in multi-data source synchronization. It also provides clear and conflict-free data for subsequent version number continuity verification.

[0123] After processing aggregate sets with identical version conflicts, perform version number continuity verification using the following formula:

[0124] ;

[0125] in, For version verification results, For data units The version number, k represents the numerical value corresponding to the version number.

[0126] When version numbers are not consecutive ( If the value is 0, the missing version must be filled in first. Based on the unconfirmed interval snapshot version number generated during the main channel switch in step S2, the missing version range is located by associating it with the data index of step S3 (e.g., the specific version missing from the minimum version number to the maximum version number of the aggregate set). Then, the data unit corresponding to the missing version is queried from the data to be filled in step S3 and added to the current aggregate set. If the data unit of the missing version cannot be found (e.g., data loss due to anomalies), the controlled pause mechanism is immediately triggered, generating an alarm event containing the missing version range and business impact assessment, and dispatching it to the management end to request manual intervention to fill in the missing version.

[0127] If the version numbers are consecutive ( If the changes are equal to 1, but there are mutually exclusive modifications between different versions (e.g., V1 changes inventory to 100, V2 changes inventory to 105, V3 changes inventory to 110), or there are no multiple version modifications (only V3 has modifications), then the latest version V3's inventory of 110 can be directly written to the target end. If the modifications in different versions are non-mutually exclusive field modifications (e.g., V1 changes inventory, V2 changes product name, V3 changes unit price), or perform compatible incremental calculations on the same numeric field (e.g., V1 changes inventory +10, V2 changes inventory -5, V3 changes inventory +3), then they need to be integrated according to the merging rules: Numeric fields are merged according to the incremental formula, using the latest version's numeric field value as the base (the latest timestamp version), and the incremental changes of lower versions are added, such as V3's inventory + V2's inventory + V1's inventory = 8; Non-mutually exclusive fields (inventory + name + unit price): based on the latest version (the latest timestamp version), the modifications of lower versions to other fields are added (e.g., V3's unit price + V2's name + V1's inventory increment); after merging, a unique final version is generated and written to the target end.

[0128] Step S5: After the data to be replenished is stored in the target end, obtain the channel anomaly feature data and channel transmission parameters during the transmission process between the source end and the target end, and input them into the deep deterministic policy gradient model based on entropy regularization. The optimal channel transmission parameters are output, and the data transmission channel is optimized based on the optimal channel parameters.

[0129] The logs generated from each channel switch, data recovery, and conflict resolution are structured and archived, and anomaly features are extracted. =[ , , ].

[0130] The state space of a deep deterministic policy gradient model based on entropy regularization under window t. , =[ , , , ],in, This refers to the abnormal feature vector (i.e., abnormal feature data) under window t. Let be the performance index vector under window t. Let be the health status vector under window t. Let be the environment context vector for window t. The action space is... ,in The action space under time window t, This is the adjustment amount for the channel switching threshold. Adjustment of shadow synchronization sampling ratio The priority revision coefficient depends on the rearrangement.

[0131] It should be noted that abnormal feature vectors By analyzing various abnormal patterns generated during operation, a seven-dimensional indicator was constructed, including: Geographic link jitter frequency: Under event window t, for all data transmission channels within the same geographical region (divided according to source / destination IP), the number of latency mutation events (continuous probe packet latency difference > 50ms) was counted and normalized. , This indicates the number of channels whose current state is not "keep-alive diagnostic"; Time-period anomaly concentration: the proportion of data consistency conflict events occurring within the statistical time window t (conflicts judged as 1 in target-side consistency verification) to the total number of data units to be verified received by the target; Object type conflict density (the number of data units judged as conflicting due to the same primary key and version number but different field values ​​ / the total number of all data units to be verified); Field conflict frequency (the frequency of conflicts in core business fields such as amount and status fields during conflict adjudication); Retry backlog coefficient (calculated based on the channel's retry logbook, representing the proportion of backlogged retry tasks); Dependency violation rate (the proportion of paths violating defined dependency edge weights in a DAG during data backup; violating defined dependencies means that the actual execution order of data backup is inconsistent with the dependency relationship defined in the DAG. For example, if node A strongly depends on node B, but A is executed before B during backup, then strong dependency is violated); and Version continuity anomaly (the proportion of missing versions detected in target-side version number continuity verification). Performance Metric Vector Used to quantify real-time operational performance, it includes four key metrics: Switching latency improvement rate: (Average historical channel switching time - Current channel switching time) / Average historical channel switching time; Data recovery success rate: Number of successfully recovered data units / Number of data units to be recovered; Conflict resolution efficiency: 1 / Average historical conflict resolution time; Resource utilization: Total number of CPU cores currently used by all channels / Ratio of bandwidth to total resources (Total number of CPU cores). Health status vector. It includes three core dimensions: Overall Health: 1 - (Number of abnormal events in the past hour / Total number of events in the past hour), where abnormal events include: channel health below the threshold, consistency verification conflict (Conflict=1), and patching failure. The closer the value is to 1, the more stable it is; Channel Stability: Number of available channels / Total number of channels, where the number of available channels is the number of channels with a health ≥ 0.8; Data Consistency Rate: (Number of data units that are completely consistent between the target and source ends / Total number of synchronized data units).

[0132] A deep deterministic policy gradient model based on entropy regularization uses an Actor network (policy network) to determine the policy gradient based on the current state. Output continuous action The core execution logic is as follows:

[0133] Channel switching threshold adjustment:

[0134] ;

[0135] in, The threshold for switching to a new channel. The original channel switching threshold, This is the adjustment amount for the channel switching threshold. This refers to the frequency of regional link jitter. For channel stability, This is the Sigmoid function.

[0136] It should be noted that the channel switching threshold adjustment is an adaptive optimization for the dynamic network environment: it uses a deep deterministic strategy gradient model to analyze abnormal feature data in real time (such as the frequency of regional link jitter and the concentration of abnormalities in a given time period) and outputs the adjustment amount. By dynamically adjusting the health threshold (becoming an adaptive value), a balance is struck between handover efficiency and stability. This avoids unnecessary handovers triggered by normal network fluctuations while ensuring timely response in the event of a real failure (preventing data loss).

[0137] Shadow synchronization sampling rate adjustment:

[0138] ;

[0139] in, , , The shadow synchronization sampling rate is the learning parameter for the policy network in the deep deterministic policy gradient algorithm. This represents the concentration of abnormalities over a given period. To improve the switching latency, For overall health.

[0140] After adjusting the sampling rate The constraints are within a reasonable range.

[0141] Dependency rearrangement priority revision:

[0142] ;

[0143] in, , , , This refers to the parameter learning for the dependency rearrangement priority of the policy network in the deep deterministic policy gradient algorithm. This is the retry packing factor. To rely on the violation rate, To improve the efficiency of conflict resolution, For data consistency and efficiency.

[0144] The revised priority score has been passed. , This represents the output score of the graph convolutional network model.

[0145] The deep deterministic policy gradient algorithm based on entropy regularization calculates reward values ​​through a Critic network (value network). The formula for the reward function in guided policy network learning is:

[0146] ;

[0147] in, For performance rewards, For consistency rewards, Rewards for efficiency As a cost penalty.

[0148] The formula for calculating performance bonuses is:

[0149] ;

[0150] in, For performance rewards, For the switching delay, To improve the success rate, For the sake of adjudication efficiency, To switch the latency weight, the default value is 0.5. To compensate for the success rate weighting, the default value is 0.3. The default value for decision efficiency is 0.2.

[0151] It should be noted that the switching delay The time interval from when the primary channel's health falls below a threshold and a switchover command is issued during time window t, until the backup channel officially takes over the data stream and successfully sends the first data unit. Recovery success rate. This represents the ratio of the number of successfully replenished data units to the total number of data units to be replenished within time window t; adjudication efficiency. This represents the average time taken for the target end to process data conflicts (i.e., the case where Conflict equals 1) within time window t, calculated as the total time taken for all conflicts divided by the number of conflicts.

[0152] The formula for calculating the consistency bonus is:

[0153] ;

[0154] in, For data consistency rate, For field conflict frequency, For version continuity anomaly rate, This is the weight for data consistency rate, with a default value of 0.5. This sets the weight for the frequency of field conflicts; the default value is 0.2. This is the weight for the version continuity anomaly rate, with a default value of 0.3.

[0155] It should be noted that data consistency rate This represents the proportion of data units that are completely identical between the target and source ends out of the total number of synchronized data units, and the frequency of field conflicts. The total number of data conflicts handled at the target end within time window t; version continuity anomaly rate. This represents the proportion of data units with non-contiguous version numbers within time window t to the total number of synchronization units.

[0156] The formula for calculating efficiency rewards is:

[0157] ;

[0158] in, This is the retry packing factor. For resource utilization, Due to data synchronization delay, The weight for the retry stacking coefficient is 0.4 by default. This is the weight for resource utilization, with a default value of 0.4. This is the weight for data synchronization delay, with a default value of 0.2.

[0159] It should be noted that the retry packing factor This represents the proportion of unprocessed retries in the channel out of the total number of retries; resource utilization rate. The ratio of system resources (CPU, bandwidth) used by the data transmission channel to the total available resources; data synchronization delay. The total time taken for a data unit to be sent from the source to the target for persistence.

[0160] The formula for calculating cost penalty is:

[0161] ;

[0162] in, As a cost penalty, This is the adjustment amount for the channel switching threshold. Adjustment of shadow synchronization sampling ratio To adjust the priority of the reordering, The weight for adjusting the threshold is set to 0.4 by default. Adjust the weight of the shadow synchronization sampling rate; the default value is 0.3. The weight for the deviation of the rearrangement revision coefficient is 0.3 by default.

[0163] Introducing policy entropy H( The degree of strategy exploration is measured by the following formula:

[0164] ;

[0165] Among them, H( ) represents the policy entropy. This refers to the policy distribution output by the policy network.

[0166] The objective function of the value network is optimized as follows:

[0167] ;

[0168] in, The objective function of the value network is... This is the discount factor, which defaults to 0.95. This is the entropy weight, which defaults to 0.05.

[0169] Each interaction ( , , , The data is stored in the experience pool, and random samples are used to train the strategy-value network to avoid the impact of sample correlation on training stability. The target network parameters are slowly updated using a soft update strategy to improve training convergence; when the reward value... The policy entropy H( ) remains stable within the range of [0.8, 1.0] for 100 consecutive time windows. Once the value stabilizes at [0.1, 0, 3], and the core indicators from step S2 to step S4 (switching latency reduced by ≥20%, recovery success rate increased by ≥15%, conflict resolution error rate reduced by ≥10%) are met, training stops, and the optimal strategy is output.

[0170] The optimal action parameters output by the deep deterministic strategy gradient algorithm are loaded into the shadow synchronization channel of S2. Only the logic of the shadow channel takes effect, while the main channel continues to use the original strategy. Through the shadow channel technology, 1% of the real business traffic from the main channel is extracted and injected into the backup channel to verify the improvement effect of the optimization strategy (such as switching misjudgment rate, recovery blocking rate, and data consistency rate). If the verification results meet the standards (such as an anomaly rate reduction of ≥30% after strategy optimization), the optimal parameters are fully updated to the global strategy configuration library and applied to steps S2 to S4. If the standards are not met, the process returns to the "anomaly feature extraction" stage to retrain the model until the strategy meets the requirements.

[0171] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, the phrase "comprising an element defined as..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0172] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for multi-data-source data synchronization, characterized in that: Includes the following steps: Step S1: Perform bandwidth evaluation and topology testing on the data transmission channel between the source and target ends to obtain channel characteristic data; based on the channel characteristic data, calculate the channel priority score, and divide the data transmission channel into the main channel and the backup channel according to the channel priority score; Step S2: Input the channel feature data into the long short-term memory network model based on the improved white whale optimization algorithm, and output the channel feature prediction value. Based on the channel feature prediction value, calculate the channel health. When the channel health of the main channel is less than the preset threshold, the data transmission is switched from the main channel to the backup channel. Step S3: Obtain the data to be supplemented after converting the main channel to the backup channel. Construct a directed acyclic graph (DAG) for supplementation based on the data to be supplemented, and input it into the graph convolutional network model. Output the supplementation priority score. Perform data supplementation based on the supplementation priority score, including the following steps: Obtain the data to be restored generated when the primary channel is converted to the backup channel; collect transaction logs and obtain the reference dependencies of each data unit in the data to be restored through the transaction logs. The reference dependencies include strong dependencies and weak dependencies. Based on the data units to be backed up and their reference dependencies, a backed-up directed acyclic graph (DAG) is constructed: Each node in the backed-up DAG represents a data unit to be backed up, and the node feature vector includes data size, business criticality level, timestamp, and channel source; each edge represents a reference dependency relationship, with the direction from the referenced data unit to the referencing data unit; the edge weight is quantized and assigned based on the dependency type: high weight 0.9 corresponds to strong dependency; medium weight 0.6 corresponds to weak dependency; low weight 0.1 corresponds to data units without direct reference but with a time sequence relationship. Based on the backfill priority score output by the graph convolution network model, the dependency rearrangement priority revision is performed according to the dependency rearrangement priority revision coefficient , combined with the dependency type of each data unit of the backfill directed acyclic graph, for the dependent node in the strong dependency relationship with an edge weight of 0.9, the original backfill priority score output by the graph convolution network model is multiplied by the dependency rearrangement priority revision coefficient ; for the weak dependency node with an edge weight of 0.6 and the time sequence correlation node with an edge weight of 0.1, the backfill priority score is kept unchanged; and finally the revised backfill priority score is generated. Based on the revised backfill priority score, a backfill plan is generated and executed: data units are sorted in descending order of backfill priority score, and data backfill is performed according to the sorted order of the data units; Step S4: After receiving the data to be replenished, the target end performs consistency verification on the data to be replenished. After the consistency verification is successful, the data to be replenished is stored in the target end.

2. The data synchronization method for multiple data sources according to claim 1, characterized in that: The process of performing bandwidth evaluation and topology testing on the data transmission channel between the source and target ends to obtain channel characteristic data includes the following specific steps: Probe packets are continuously sent to the data transmission channel between the source and the target to obtain the channel feature vector of each data transmission channel. The channel feature vector includes: round-trip delay, packet loss rate and throughput, transmission queue backlog depth and retry event density.

3. The data synchronization method for multiple data sources according to claim 2, characterized in that: The calculation of channel priority scores based on channel feature data includes the following steps: Based on channel feature data, latency score, packet loss rate score, and throughput score are calculated, thereby calculating channel priority score: ； in, Score the channel priority. To delay scoring, To score the packet loss rate, For throughput score, For delayed score weights, As the weight for packet loss rate score, Weights are assigned to throughput scores.

4. The data synchronization method for multiple data sources according to claim 3, characterized in that: The process of inputting channel feature data into a long short-term memory network model based on an improved beluga optimization algorithm and outputting predicted channel feature values ​​includes the following steps: Population initialization based on reverse learning: half of the original random solutions X are randomly generated, covering the hyperparameter search range of the Long Short-Term Memory (LSTM) network; then, the reverse solution X' is calculated for each original random solution X. The original random solutions and reverse solutions are merged, and the solution with the best fitness is selected by the fitness function to form the final initial population. The attraction mechanism of the firefly algorithm is introduced: in the exploration phase of the algorithm, for any two solutions in the population, the magnitude of the attraction β is related to the fitness difference and Euclidean distance between individuals. Solutions with poor fitness will move towards solutions with better fitness. The magnitude of this movement is determined by both the attraction factor β and the random perturbation term, as shown in the formula: ； in, This represents the position of the m-th individual in the population during the dd-th iteration. This represents the updated position of the m-th individual in the (dd+1)-th iteration. This indicates that in the dd-th iteration, the fitness of the population is better than... The position of the m2th individual, bc is the step size factor which defaults to 0.05, and rand(0,1) is a random number between 0 and 1; By calculating the variance of population fitness, if the variance is less than a preset threshold, the population is deemed to have insufficient diversity. Then, for the current optimal solution... Generating nonlinear inverse solutions based on lens imaging principles '; Population update: The generated reverse solution Replace the worst-fitting solution in the population, recalculate the population fitness, break the local optimum constraint, and increase the probability of finding the global optimal hyperparameter combination; after the population update is completed, the algorithm continues to enter the next iteration, repeating the optimization process of the exploration and development stage until the preset maximum number of iterations is reached, and finally the hyperparameter-optimized long short-term memory network model is obtained. Input the channel feature data [TZ(1),TZ(2),TZ(3),...,TZ(t)] of t time windows into a trained long short-term memory network model based on the improved white whale optimization algorithm, and output the predicted channel feature values ​​for the next t+1 time windows. , =[ , (t+1) (t+1) (t+1) [(t+1)], where, This is the predicted packet loss rate for time window t+1. (t+1) represents the predicted round-trip delay within the time window t+1. (t+1) represents the predicted value of the sending queue backlog depth under time window t+1. (t+1) represents the predicted retry event density within time window t+1. (t+1) represents the predicted throughput value within the time window t+1.

5. The data synchronization method for multiple data sources according to claim 4, characterized in that: The channel health status is: ； in, Here, t represents the channel health score, T represents the time window index, and c represents the time window length threshold. This represents the total number of features in the channel feature data. This represents the predicted value of the c-th feature in the channel feature data within the time window t+1. This represents the true value of the c-th feature in the channel feature data within the time window t+1. To prevent division by zero constant, the default value is 0. .

6. The data synchronization method for multiple data sources according to claim 5, characterized in that: After receiving the data to be replenished, the target end performs consistency verification on the data to be replenished, including the following specific steps: After receiving the data to be replenished, the target end aggregates all channel data units according to the primary key of the data unit in the data to be replenished. Check if there are data units in the aggregate collection with the same version number and the same field value in the data unit. When there is duplicate data transmission, use the data unit with the latest timestamp as the only valid unit of that version and remove duplicates; then check if there are different modifications to the same field in the same version number. When there are conflicts between fields of the same version, calculate the adjudication score for each data unit using the following formula: ； in, For data units The score of the ruling, As a weight for source credibility, As a business criticality weight, Indicates the arrival timestamp of the data unit. This represents the maximum value of the arrival timestamps within the aggregate set. , , For strategy weights, + + =1; choose The highest-valued data unit is designated as the sole valid unit for that version, and all other data units in that version are removed. After processing aggregate sets with identical version conflicts, perform version number continuity verification using the following formula: ； in, For version verification results, For data units The version number, k represents the numerical value corresponding to the version number.

7. A data synchronization method for multiple data sources according to claim 6, characterized in that: It also includes step S5: After the data to be replenished is stored in the target end, the channel anomaly feature data and channel transmission parameters during the transmission process between the source end and the target end are obtained and input into the deep deterministic policy gradient model based on entropy regularization. The optimal channel transmission parameters are output and the data transmission channel is optimized based on the optimal channel parameters.

8. A data synchronization system for multiple data sources, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

Images