An automated transformation and load integration method supporting multi-source data synchronization

By automatically generating scheduling tasks and dynamically adjusting the number of nodes, combined with data source connectors, data quality assessment, and virtual mirror environment, the problem of uneven task allocation for multi-source data synchronization in MES is solved, achieving efficient resource utilization and system stability, and improving the compatibility and security of data processing.

CN121032155BActive Publication Date: 2026-01-23JIANGSU DAODA INTELLIGENT TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202511563700.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-10-30
Publication Date
2026-01-23
Estimated Expiration
2045-10-30

AI Technical Summary

Technical Problem

Existing multi-source data synchronization methods cannot achieve dynamic optimal allocation of tasks in Manufacturing Execution Systems (MES), resulting in inefficient utilization of computing resources, poor system stability, insufficient compatibility and scalability, and an inability to meet both real-time and periodic analysis requirements, leading to performance bottlenecks and consistency issues.

Method used

The system employs automatic task generation and scheduling, dynamically adjusts task allocation and node quantity, acquires data in real time through data source connectors and performs real-time data extraction and processing, uses a data quality assessment model to identify data quality, constructs a virtual mirror environment for simulation calculations, dynamically generates optimal write statements, performs data consistency checks, monitors task execution to optimize execution order, and achieves elastic scheduling and efficient resource utilization.

Benefits of technology

It achieves efficient utilization of computing resources and system stability, has good compatibility and scalability, can adapt to multi-source data environments, takes into account both real-time and periodic analysis needs, improves data credibility, avoids performance bottlenecks and consistency issues, and enhances the security and controllability of the loading process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121032155B_ABST
    Figure CN121032155B_ABST
Patent Text Reader

Abstract

The application discloses an automatic conversion and loading integrated method supporting multi-source data synchronization, and belongs to the field of data management, and the integrated method specifically comprises the following steps: I, after starting the process, automatically generating a scheduling task, distributing each subtask to an idle task executor node by a scheduling center, and dynamically adjusting task distribution and the number of nodes; the application realizes dynamic optimal distribution of tasks, guarantees efficient utilization of computing resources and system stability, can adapt to different multi-source data environments, has good compatibility and expansibility, simultaneously considers real-time and periodic analysis requirements, effectively identifies noise data, improves data credibility, avoids performance bottlenecks and consistency problems in advance, improves the safety and controllability of the loading link, guarantees data integrity and traceability, and enhances system stable operation capability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data management, and in particular to an automated transformation and loading integration method that supports multi-source data synchronization. Background Technology

[0002] In Manufacturing Execution System (MES) applications, data is a key element driving production process monitoring, quality traceability, and decision optimization. With the continuous development of smart manufacturing, data sources on the production floor are becoming increasingly diverse, encompassing MES real-time databases, MES historical databases, Warehouse Management Systems (WMS), and Enterprise Performance Management Systems (EPM). This multi-source data is heterogeneous (structured, semi-structured, and unstructured data coexist) and dynamic (real-time and historical data in parallel), while the production environment demands high real-time performance and accuracy from data processing. Traditional data processing technologies mainly rely on ETL (Extract-Transform-Load) frameworks, single data source connectors, and simple data synchronization mechanisms (such as timed polling and triggered acquisition). These methods can partially solve data migration and update problems, but due to a lack of overall architectural design and deep integration, they are difficult to adapt to the multi-source, multi-dimensional, and highly time-sensitive data processing needs of MES scenarios. Therefore, inventing an automated transformation and loading integration method that supports multi-source data synchronization has become particularly important.

[0003] Existing automated transformation and loading integration methods that support multi-source data synchronization cannot perform dynamic optimal allocation of tasks, cannot guarantee efficient utilization of computing resources and system stability, have poor compatibility and scalability, and cannot meet the needs of real-time and periodic analysis. They also cannot avoid performance bottlenecks and consistency issues, reducing the security and controllability of the loading process. To address these issues, we propose an automated transformation and loading integration method that supports multi-source data synchronization. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an automated conversion and loading integration method that supports multi-source data synchronization.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An automated transformation and loading integration method supporting multi-source data synchronization is provided, and the specific steps of the integration method are as follows:

[0007] Ⅰ. After the process starts, scheduling tasks are automatically generated. The scheduling center distributes each subtask to idle task executor nodes and dynamically adjusts task allocation and the number of nodes.

[0008] II. After receiving the scheduling instruction, the task executor automatically calls the corresponding data source connector according to the task description, and at the same time builds a real-time data extraction and processing pipeline.

[0009] III. The extracted data of various types are temporarily stored in the raw data cache area, and the quality of each data is identified through the data quality assessment model before the processing flow.

[0010] IV. Dynamically adjust the data preprocessing strategy based on the recognition results, and then pull various types of preprocessed data according to the rule engine and perform transformation processing;

[0011] V. Construct a virtual image environment for the target system, simulate and calculate the various types of data to be written, and predict their impact on the system state;

[0012] VI. Based on the target storage structure, dynamically generate the optimal write statement for the various types of data that have passed the verification, and perform data consistency verification after the write operation is executed;

[0013] VII. During the scheduling task cycle, continuously monitor the task execution status and dynamically optimize the task execution order based on the urgency of the business, the freshness of the data, and the system load.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] This invention periodically collects resource metrics from each task executor node, calculates a comprehensive load index, dynamically senses the overall load distribution, and allocates tasks to the node with the lowest load when they arrive. When the average load exceeds the scaling-up or scaling-down threshold, the number of nodes is automatically adjusted for elastic scheduling. Then, the executor calls the corresponding data source connector according to the task description, acquires data in real time, marks it with a timestamp, and writes it to the event buffer. Subsequently, the stream processing engine performs logical time sorting and windowed aggregation, outputting real-time incremental streams or periodic batch results. At the same time, it monitors data access pressure, processing latency, and synchronization latency. If the loss rate exceeds the threshold, a backtracking mechanism is triggered to retrieve data from the cache or logs. Each node locally extracts field integrity, rationality, and consistency features to train a lightweight linear model for quality prediction. Local model parameters are homomorphically encrypted and uploaded to the federated parameter server for global aggregation. After convergence, the data is distributed to each node, and the quality of each data point is evaluated and anomalies are identified in real time. Before loading, the system constructs a virtual image using a digital twin mechanism, calculates resource consumption and conflict rate of the data to be written, and generates loading risk indicators. If the risk is below the threshold, real writing is allowed; if it exceeds the threshold, an alert is triggered and the strategy is optimized to achieve dynamic optimal allocation of tasks, ensuring efficient utilization of computing resources and system stability. The system can adapt to different multi-source data environments, has good compatibility and scalability, and takes into account both real-time and periodic analysis needs. It effectively identifies noisy data, improves data credibility, avoids performance bottlenecks and consistency issues in advance, improves the security and controllability of the loading process, ensures data integrity and traceability, and enhances the system's stable operation capability. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0017] Figure 1 This is a flowchart of an automated conversion and loading integration method for supporting multi-source data synchronization proposed in this invention. Detailed Implementation

[0018] Reference Figure 1 An automated transformation and loading integration method supporting multi-source data synchronization is described, with the following specific steps:

[0019] Ⅰ. After the process starts, scheduling tasks are automatically generated. The scheduling center distributes each subtask to idle task executor nodes and dynamically adjusts task allocation and the number of nodes.

[0020] II. After receiving the scheduling instruction, the task executor automatically calls the corresponding data source connector according to the task description, and at the same time builds a real-time data extraction and processing pipeline.

[0021] III. The extracted data of various types are temporarily stored in the raw data cache area, and the quality of each data is identified through the data quality assessment model before the processing flow.

[0022] IV. Dynamically adjust the data preprocessing strategy based on the recognition results, and then pull various types of preprocessed data according to the rule engine and perform transformation processing;

[0023] V. Construct a virtual image environment for the target system, simulate and calculate the various types of data to be written, and predict their impact on the system state;

[0024] VI. Based on the target storage structure, dynamically generate the optimal write statement for the various types of data that have passed the verification, and perform data consistency verification after the write operation is executed;

[0025] VII. During the scheduling task cycle, continuously monitor the task execution status and dynamically optimize the task execution order based on the urgency of the business, the freshness of the data, and the system load.

[0026] Once the process begins, scheduled tasks are automatically generated. The scheduling center distributes each subtask to idle task executor nodes and dynamically adjusts task allocation and the number of nodes. Specifically:

[0027] S1.1: The system periodically collects the resource indicators of each task executor node and calculates the comprehensive load index of each task executor node based on the collected resource indicators to obtain the overall load distribution. When a new task arrives, the scheduling center automatically assigns the new task to the task executor node with the lowest comprehensive load index.

[0028] S1.2: Calculate the average load of the task executor nodes. If the average load of the nodes is higher than the preset expansion threshold, expansion is triggered, the number of new nodes is determined, and the average load is restored to the preset range. If the average load of the nodes is lower than the preset shrinkage threshold, and the number of nodes is greater than the minimum limit, shrinkage is triggered, the number of nodes to be reduced is determined, and excess nodes are released, and the average load is restored to the preset load threshold.

[0029] S1.3: After the expansion or reduction is completed, the system continuously monitors the new load situation and recalculates the average load of the task executor node cluster, entering a new round of adjustment cycle. At the same time, the weight parameters of each resource indicator and the load threshold are adjusted according to historical monitoring data.

[0030] It should be further explained that resource metrics specifically include CPU utilization, memory usage, and task queue length, etc.

[0031] The specific calculation formula for the comprehensive load index mentioned in S1.1 is as follows:

[0032] ;

[0033] In the formula, Representing the The overall load index of each task executor node; Representing the CPU utilization of each task executor node; Representing the Memory usage of each task executor node; Representing the Normalized value of the task queue length of each task executor node; , as well as These represent the weight parameters for each resource indicator, and the importance of different indicators can be adjusted according to business needs.

[0034] The specific calculation formula for the average load mentioned in S1.2 is as follows:

[0035] ;

[0036] In the formula, This represents the average load across all task executor nodes. Represents the total number of current task executor nodes;

[0037] The specific formula for calculating the number of new nodes mentioned in S1.2 is as follows:

[0038] ;

[0039] In the formula, Represents the number of newly added nodes; Represents the preset load threshold; Represents the floor function;

[0040] The specific calculation formula for reducing the number of nodes as described in S1.2 is as follows:

[0041] ;

[0042] In the formula, This represents reducing the number of nodes; This represents the floor function.

[0043] After receiving the scheduling instruction, the task executor automatically calls the corresponding data source connector according to the task description, and at the same time builds a real-time data extraction and processing pipeline.

[0044] Specifically, the task executor automatically calls the corresponding connector module based on the data source type parameter in the task description, and calculates the data arrival rate of various types of data in real time. Based on the real-time data arrival rate, it dynamically senses the data access pressure of different sources. The system timestamps the various types of data that are accessed in real time and writes them into the event buffer. At the same time, it sorts and processes the data according to the logical time of each type of data. After processing, it calculates the processing latency of each type of data, sets a rolling time window of length W, divides the continuous data stream into multiple time windows, and aggregates and averages the data within each time window. Then, the data after time windowing is transmitted directly to the target storage as a real-time incremental data stream according to the usage requirements, or is periodically batch-fed to disk to form batch processing results for synchronization processing. After processing, it calculates the synchronization latency of a complete synchronization operation. During data synchronization, it calculates the data loss rate of each type of data in real time. If the data loss rate is higher than the preset loss threshold, it initiates a backtracking mechanism to retrieve data from the original data cache or source logs. It records the data access pressure, processing latency, and synchronization latency of each round of real-time data extraction and processing pipeline.

[0045] In addition, it should be noted that the specific formula for calculating the data arrival rate is as follows: ;

[0046] In the formula, Representative from data source Data arrival rate; Represents the time interval Internally, from the data source The number of events received; Represents the sampling time interval;

[0047] The specific formula for calculating the aggregate average is as follows: ;

[0048] In the formula, Represents the rolling time window The average value of internal event attribute values; Represents the rolling time window The number of events within, ; Representative event The attribute value; This indicates that it belongs to the rolling time window. A collection of events;

[0049] The specific formula for calculating the data loss rate is as follows:

[0050] ;

[0051] In the formula, Represents the data loss rate; This represents the theoretical number of events that should be received. This represents the actual number of events received.

[0052] The connector module specifically includes relational database connectors, message queue connectors, API interface connectors, etc.; the time window specifically includes scrolling windows, sliding windows, and session windows.

[0053] The extracted data are temporarily stored in the raw data cache area, and the quality of each data is identified through a data quality assessment model before the processing flow.

[0054] Specifically, each task executor node performs preliminary feature extraction on the raw data locally, obtaining information on the completeness of fields, the rationality of values, and the consistency of distribution for various types of raw data. The extracted information is then combined as the quality features of the corresponding raw data. Each task executor node initializes a lightweight linear model, sets its loss function, collects historical data quality features and corresponding actual quality labels, and then divides the historical quality features into training and validation sets. The training set is further divided into multiple batches of training samples, which are sequentially input into the lightweight linear model. The lightweight linear model then outputs predicted quality labels for various types of data using the Sigmoid function. The loss value between the predicted and actual quality labels is calculated using the loss function. Based on the backpropagation algorithm, the loss value is input into the lightweight linear model, and the model parameters are updated using gradient descent. After each training round, the validation set is input into the lightweight linear model, and the model performance is evaluated. Yes, if the model performance meets the preset threshold, training stops; otherwise, training continues until the model performance reaches the preset threshold. After local training, each task executor node encrypts its local parameters using homomorphic encryption and uploads them to the federated parameter server. After receiving the encrypted parameters from all participating task executor nodes, the federated parameter server homomorphically aggregates the encrypted local parameters to obtain the latest global parameters. The new global parameters are then distributed to each task executor node. Each node uses these global parameters as initialization and enters the next round of local training until the changes in the global parameters after multiple rounds of training converge to a preset range. Training then stops, and the final global parameters are sent to each task executor node, replacing the original local parameters of the lightweight linear model in each task executor node. Afterward, the quality features of the latest raw data are input into the lightweight linear model, and anomaly scores for various types of data are output. If the anomaly score exceeds a set threshold, the data is judged as abnormal or noise.

[0055] It should be further explained that the specific calculation formula for the lightweight linear model is as follows:

[0056] ;

[0057] In the formula, Representative model for the first Predicted output for group data; This represents the Sigmoid function, which maps the output to the (0,1) interval; Represents the weight vector; Representing the Feature vectors of a set of data; Represents the bias term;

[0058] The specific formula for calculating the loss function is as follows:

[0059] ;

[0060] In the formula, This represents the average loss function value; Represents the total number of samples participating in the training; Representing the The actual quality label of the group data;

[0061] The specific calculation formula for homomorphic aggregation is as follows:

[0062] ;

[0063] In the formula, Representative at the The global model parameter set generated in the round; This represents the number of task executor nodes participating in this round of training; Representative node In the Wheel parameters.

[0064] Reference Figure 1 An automated transformation and loading integration method supporting multi-source data synchronization is described, with the following specific steps:

[0065] The data preprocessing strategy is dynamically adjusted based on the recognition results. Then, the preprocessed data of various types is pulled by the rule engine and transformed.

[0066] A virtual mirror environment of the target system is constructed, and various types of data to be written are simulated and calculated to predict their impact on the system state.

[0067] Specifically, based on the metadata of the current target database or storage engine, a structurally consistent mirror copy is generated in the virtual mirror environment. According to the logical write requirements of each data set, the batch data to be written is converted into a load vector. Each load vector is input to the mirror copy, which performs calculations on the batch data to be written and calculates the potential resource consumption of each data set in the mirror system. Simultaneously, in the virtual write simulation, the data conflict rate is calculated in real time. If the data conflict rate exceeds a preset conflict threshold, it indicates a serious consistency problem in the batch data, and the data is marked for processing. Based on the simulation results, loading risk indicators for various types of data are calculated. Based on these loading risk indicators, the mirror copy comprehensively judges the loading feasibility of various types of data. If the loading risk indicator is below the safety threshold, real data writing is allowed; if the loading risk indicator exceeds the safety threshold, an alert is triggered and the write strategy is adjusted.

[0068] It should be noted that the specific formula for calculating resource consumption is as follows:

[0069] ;

[0070] ;

[0071] In the formula, This represents the predicted CPU resource consumption. This represents the total number of data to be written. Representing the The computational cost coefficient corresponding to each data item during processing; Representing the The logical write requirement for each data item; This represents the predicted disk I / O consumption. Representing the I / O consumption coefficient for each data item;

[0072] The specific formula for calculating the data conflict rate is as follows:

[0073] ;

[0074] In the formula, Represents the predicted conflict rate; This represents the number of conflicting data points detected during the simulation.

[0075] The specific formula for calculating the loading risk index is as follows:

[0076] ;

[0077] In the formula, Represents loading risk indicators; This represents the maximum available CPU resources of the target system; This represents the maximum available disk I / O of the target system; , as well as These represent the risk weighting coefficients for each indicator, set according to the system's tolerance level.

[0078] The metadata of a storage engine specifically includes table structure, indexing rules, transaction mechanisms, etc.

[0079] Based on the verified data, the optimal write statement is dynamically generated according to the target storage structure, and data consistency verification is performed after the write operation is executed.

[0080] During the scheduling task cycle, the task execution status is continuously monitored, and the execution order of tasks is dynamically optimized based on the urgency of business, data freshness, and system load.

Claims

1. An automated conversion and loading integration method supporting multi-source data synchronization, characterized in that, The specific steps of this integration method are as follows: I. After the process begins, scheduling tasks are automatically generated. The scheduling center distributes each subtask to idle task executor nodes and dynamically adjusts task allocation and the number of nodes. The specific steps for dynamically adjusting task allocation and the number of nodes are as follows: S1.1: The system periodically collects the resource indicators of each task executor node and calculates the comprehensive load index of each task executor node based on the collected resource indicators to obtain the overall load distribution. When a new task arrives, the scheduling center automatically assigns the new task to the task executor node with the lowest comprehensive load index. S1.2: Calculate the average load of the task executor nodes. If the average load of the nodes is higher than the preset expansion threshold, expansion is triggered, the number of new nodes is determined, and the average load is restored to the preset range. If the average load of the nodes is lower than the preset shrinkage threshold, and the number of nodes is greater than the minimum limit, shrinkage is triggered, the number of nodes to be reduced is determined, and excess nodes are released, and the average load is restored to the preset load threshold. S1.3: After the expansion or reduction is completed, the system continues to monitor the new load situation and recalculates the average load of the task executor node cluster, entering a new round of adjustment cycle. At the same time, the weight parameters of each resource indicator and the load threshold are adjusted according to historical monitoring data. II. After receiving the scheduling instruction, the task executor automatically calls the corresponding data source connector according to the task description, and at the same time builds a real-time data extraction and processing pipeline. III. The extracted data of various types are temporarily stored in the original data cache area, and the quality of each data type is identified through a data quality assessment model before the processing flow. The specific steps for identifying the quality of each data type through the data quality assessment model are as follows: S3.1: Each task executor node performs preliminary feature extraction on the raw data locally, obtains information on the field completeness, value rationality, and distribution consistency of various types of raw data, and combines the extracted information on various indicators as the quality features of the corresponding raw data. S3.2: Each task executor node initializes a set of lightweight linear models, sets the loss function of the lightweight linear model, collects the quality features of historical data and the corresponding actual quality labels, and then divides the historical quality features into training set and validation set. S3.3: Divide the training set data into multiple batches of training sample sets and input them into the lightweight linear model in sequence. Then, the lightweight linear model outputs the predicted quality labels of various data through the Sigmoid function and calculates the loss value between the predicted quality labels and the corresponding actual quality labels through the loss function. S3.4: Based on the backpropagation algorithm, the loss value is input into the lightweight linear model, and the model parameters are updated by gradient descent. After each round of training, the validation set is input into the lightweight linear model, and the model performance is evaluated. If the model performance meets the preset threshold, training is stopped; otherwise, training continues until the model performance reaches the preset threshold. S3.5: After each task executor node completes its local training, it encrypts its local parameters using homomorphic encryption and uploads them to the federated parameter server. After receiving the encrypted parameters from all participating task executor nodes, the federated parameter server performs homomorphic aggregation on each encrypted local parameter to obtain the latest global parameters. S3.6: Distribute the new global parameters to each task executor node. Each node will use the global parameters as initialization and enter the next round of local training until the changes in the global parameters after multiple rounds of training converge to the preset range, and then stop training. S3.7: The final global parameters are sent to each task executor node, and the original local parameters of the lightweight linear model in each task executor node are replaced. Then, the quality features of the latest raw data are input into the lightweight linear model, and the abnormal scores of various types of data are output. If the abnormal score exceeds the set threshold, the data is judged as abnormal or noise. IV. Dynamically adjust the data preprocessing strategy based on the recognition results, and then pull various types of preprocessed data according to the rule engine and perform transformation processing; V. Construct a virtual mirror environment for the target system, simulate and calculate the various types of data to be written, and predict the impact of these data types on the system state. The specific steps for predicting the impact of these data types on the system state are as follows: S4.1: Based on the metadata of the current target database or storage engine, generate a mirror copy with the same structure in the virtual mirror environment, and convert the batch data to be written into a load vector according to the logical write requirements of each data. S4.2: Input each load vector to the mirror replica. The mirror replica performs calculations on the batch data to be written and calculates the resource consumption that each group of data may generate in the mirror system. At the same time, in the virtual write simulation, the data conflict rate is calculated in real time. If the data conflict rate exceeds the preset conflict threshold, it indicates that there is a serious consistency problem in the batch data and it is marked for processing. S4.3: Based on the simulation results, calculate the loading risk index of various types of data. Based on each loading risk index, the mirror copy comprehensively judges the loading feasibility of various types of data. If the loading risk index is lower than the safety threshold, real data writing is allowed. If the loading risk index exceeds the safety threshold, an early warning is triggered and the writing strategy is adjusted. VI. Based on the target storage structure, dynamically generate the optimal write statement for the various types of data that have passed the verification, and perform data consistency verification after the write operation is executed; VII. During the scheduling task cycle, continuously monitor the task execution status and dynamically optimize the task execution order based on the urgency of the business, the freshness of the data, and the system load.

2. The automated conversion and loading integration method supporting multi-source data synchronization according to claim 1, characterized in that, The specific calculation formula for the comprehensive load index mentioned in S1.1 is as follows: ; In the formula, Representing the The overall load index of each task executor node; Representing the CPU utilization of each task executor node; Representing the Memory usage of each task executor node; Representing the Normalized value of the task queue length of each task executor node; , as well as These represent the weight parameters for each resource indicator, and the importance of different indicators can be adjusted according to business needs. The specific calculation formula for the average load mentioned in S1.2 is as follows: ; In the formula, This represents the average load across all task executor nodes. Represents the total number of current task executor nodes; The specific formula for calculating the number of new nodes mentioned in S1.3 is as follows: ; In the formula, Represents the number of newly added nodes; Represents the preset load threshold; Represents the floor function; The specific calculation formula for reducing the number of nodes as described in S1.4 is as follows: ; In the formula, This represents reducing the number of nodes; This represents the floor function.

3. The automated conversion and loading integration method supporting multi-source data synchronization according to claim 1, characterized in that, The specific steps for constructing the real-time data extraction and processing pipeline described in step II are as follows: S2.1: The task executor automatically calls the corresponding connector module according to the data source type parameter in the task description, and calculates the data arrival rate of various types of data in real time. Based on the real-time data arrival rate, it dynamically senses the data access pressure of different sources. S2.2: The system will stamp the various types of data that are received in real time with the corresponding timestamps and write them into the event buffer. At the same time, it will sort and process the data according to the logical time of each type of data. After processing, it will calculate the processing delay of each type of data. S2.3: Set a rolling time window of length W, divide the continuous data stream into multiple time windows, and aggregate and average the data in each time window. Then, according to the usage requirements, the data after time windowing is directly transmitted to the target storage as a real-time incremental data stream, or is periodically batch-written to disk to form batch processing results for synchronization processing. After the processing is completed, the synchronization delay of a complete synchronization operation is calculated. S2.4: During data synchronization, calculate the data loss rate of various types of data in real time. If the data loss rate is higher than the preset loss threshold, start the backtracking mechanism to pull data again from the original data cache or source log. Record the data access pressure, processing delay and synchronization delay of each round of real-time data extraction and processing pipeline.

4. The automated conversion and loading integration method supporting multi-source data synchronization according to claim 1, characterized in that, The specific calculation formula for the lightweight linear model described in S3.2 is as follows: ; In the formula, Representative model for the first Predicted output for group data; This represents the Sigmoid function, which maps the output to the (0,1) interval; Represents the weight vector; Representing the Feature vectors of a set of data; Represents the bias term; The specific calculation formula for the loss function described in S3.2 is as follows: ; In the formula, This represents the average loss function value; Represents the total number of samples participating in the training; Representing the The actual quality label of the group data; The specific calculation formula for homomorphic aggregation described in S3.5 is as follows: ; In the formula, Representative at the The global model parameter set generated in the round; This represents the number of task executor nodes participating in this round of training; Representative node In the Wheel parameters.

5. The automated conversion and loading integration method supporting multi-source data synchronization according to claim 1, characterized in that, The specific calculation formula for resource consumption mentioned in S4.2 is as follows: ; ; In the formula, This represents the predicted CPU resource consumption. This represents the total number of data to be written. Representing the The computational cost coefficient corresponding to each data item during processing; Representing the The logical write requirement for each data item; This represents the predicted disk I / O consumption. Representing the I / O consumption coefficient for each data item; The specific formula for calculating the data conflict rate described in S4.2 is as follows: ; In the formula, Represents the predicted conflict rate; This represents the number of conflicting data points detected during the simulation. The specific calculation formula for the loading risk index mentioned in S4.2 is as follows: ; In the formula, Represents loading risk indicators; This represents the maximum available CPU resources of the target system; This represents the maximum available disk I / O of the target system; , as well as These represent the risk weighting coefficients for each indicator, set according to the system's tolerance level.

Citation Information

Patent Citations

  • Multi-source computing power data integration and intelligent scheduling system and method

    CN118916147A