A data synchronization method and system based on front-end orchestration management
By combining dynamic threshold verification with multi-dimensional algorithms in front-end orchestration management, the problem of insufficient data synchronization adaptability is solved, achieving efficient and accurate multi-source data fusion and a balance between security and efficiency, thus adapting to the data synchronization needs of complex business scenarios.
Patent Information
- Application Number
- CN202610185041.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies lack adaptability in data synchronization, struggle to dynamically balance source reliability and business priority in multi-source data fusion, lack flexible adjustment of encryption strategies, and have weak coordination between front-end configuration and underlying execution links, making it difficult to cope with dynamic changes in requirements under complex business scenarios.
By combining dynamic threshold verification with multi-dimensional algorithms, and improving synchronization efficiency and accuracy through front-end orchestration management, a full-link collaborative mechanism is achieved. The encryption strategy is dynamically adjusted to balance security and performance, and multi-drive collaborative execution is supported.
It significantly improves the intelligence and adaptability of data synchronization, ensures strict and flexible data verification, accurate fusion of multi-source data, balances security and efficiency, lowers the threshold for use, and adapts to complex business scenarios.
Smart Images
Figure CN122093407A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data synchronization, specifically relating to a data synchronization method and system based on front-end orchestration management. Background Technology
[0002] Data synchronization updates the front-end view through real-time or timed communication mechanisms, such as WebSocket, polling, or state management libraries, ensuring the accuracy and timeliness of the information seen by the user; front-end orchestration management focuses on coordinating the execution order and dependencies of multiple components, modules, or asynchronous tasks to ensure smooth application flow and clear logic.
[0003] In existing technologies, data synchronization typically relies on fixed threshold verification, resulting in insufficient adaptability. Multi-source data fusion makes it difficult to dynamically balance source reliability and business priority. Encryption strategies lack flexible adjustment mechanisms, which can easily lead to an imbalance between security and performance. Furthermore, the coordination between front-end configuration and underlying execution links is weak, and the full-link monitoring and lifecycle management capabilities are insufficient, making it difficult to cope with dynamic changes in requirements under complex business scenarios. Summary of the Invention
[0004] To address the problems in the prior art, this invention proposes a data synchronization method and system based on front-end orchestration management. By combining dynamic threshold verification with multi-dimensional algorithms, the synchronization efficiency and accuracy are improved. The front-end orchestration and end-to-end collaborative mechanism further enhance the usability and manageability of the data synchronization method.
[0005] The technical solution of the present invention is as follows: In a first aspect, the present invention provides a data synchronization method based on front-end orchestration management, comprising the following steps: Select the data table and field information to be monitored through the front-end interface, generate monitoring configuration information, and orchestrate the synchronization process logic according to business needs. The monitoring configuration information is stored in the configuration center in real time, and real-time monitoring rules are generated based on the monitoring configuration information. According to the listening rules, the data table and field information change data are captured, and each change data is encapsulated into a standard data unit and sent to the message queue. The dynamic strategy engine is invoked to pull standard data units from the message queue and preprocess them, and the preprocessing results are pushed to the synchronization adaptation module. The synchronization adaptation module generates a synchronization instruction set by matching the corresponding driver type according to the synchronization process logic, and performs data synchronization operations based on the preprocessing results. The synchronization process is monitored across the entire chain, and task lifecycle management is performed based on the lifecycle rules configured on the front end.
[0006] Furthermore, the preprocessing specifically includes: The dynamic strategy engine pulls data verification rules, field mapping relationship tables and encryption / decryption algorithms from the configuration center, traverses the data content of standard data units field by field and matches the corresponding data verification rules. The verification rules include verifying the value range of numeric fields and verifying the format specifications of string fields. The field mapping table includes the mapping relationship between the original field structure of the current database field information and the field structure of the target database field information. For standard data units that pass the verification, based on the field mapping table, the data within the standard data units is merged, split, or converted in format. At the same time, field conflicts are handled according to the priority rules preset by the configuration center. For standard data units after field mapping is completed, the sensitive fields are queried according to the sensitive field configuration preset in the configuration center, and the encryption and decryption algorithms are called to perform encryption and decryption processing on the sensitive fields. The encryption key or decryption certificate is dynamically obtained through the security service.
[0007] Furthermore, the formula for calculating the value range of the numeric field is as follows: ; in, T current This specifies the range of values for the current numeric field. T base The baseline threshold; d This is the fluctuation sensitivity coefficient; s s The standard deviation of historical change data; S c For business scenario coefficients.
[0008] Furthermore, the specific steps for merging the fields are as follows: Based on the field reliability score, the proportion of non-empty fields, and the field business priority in the configuration center, calculate the fusion weight of each field: ; in, For the first The fusion weight of each field; This is the balance coefficient; For the first Reliability scores for each field; For the first The percentage of non-empty fields in each field; For the first Business priority of each field; n This represents the total number of fields that need to be merged.
[0009] Furthermore, the key length and number of iterations of the encryption / decryption algorithm are set through a preset encryption strength mapping table, which includes the key length and number of iterations corresponding to different comprehensive encryption strength values; The comprehensive value of encryption strength S enc The calculation formula is as follows: ; in, L b The baseline strength for the sensitivity level; m This is a risk amplification factor. V s This refers to the real-time security factor of the transmission link. This is a historical attack risk index.
[0010] Furthermore, the changes to the data tables and field information are captured based on data transmission services and data replication services; The default format of the standard data unit is JSON.
[0011] Furthermore, a preset pull frequency rule is established, which includes the pull frequency under different preprocessing load conditions; Standard data units are retrieved from the message queue according to the retrieval frequency rules.
[0012] Furthermore, an asynchronous buffering mechanism is used to send standard data units to the message queue.
[0013] Furthermore, the synchronization process is monitored across the entire link, and an alarm mechanism is triggered when an abnormal state is detected, sending the abnormal state information to the user.
[0014] Secondly, the present invention provides a data synchronization system based on front-end orchestration management, the system comprising: The front-end orchestration and configuration module is used to select the data table and field information to be monitored, generate monitoring configuration information, and store the configuration information in the configuration center in real time. The data monitoring module is used to generate real-time monitoring rules, capture changes in data tables and field information, and encapsulate them into standard data units. The message queue module is used to receive standard data units and call the dynamic strategy engine to preprocess the standard data units. The synchronization adaptation module is used to receive the preprocessing results, generate a synchronization instruction set by matching the corresponding driver type according to the synchronization process logic, and perform data synchronization operations based on the preprocessing results. The end-to-end monitoring module integrates change data capture records, preprocessing logs, and synchronization operation status to generate monitoring data; The lifecycle management module combines monitoring data and front-end configuration information to perform task lifecycle management.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: By deeply integrating a dynamic strategy engine with multi-dimensional algorithmic formulas, the intelligence and adaptability of data synchronization are significantly improved. The dynamic threshold verification formula can dynamically adjust the verification standard based on the historical fluctuation characteristics of the data source and the business scenario, avoiding the misjudgment or omission caused by traditional fixed thresholds. This allows data verification to both strictly control quality and flexibly adapt to business changes. The multi-source field fusion weight formula, by comprehensively considering the reliability of the source, field integrity, and business priority, achieves accurate fusion of data from different systems, solving the problem of multi-source data conflicts and ensuring that the target system receives high-quality data that has been optimized and integrated. The encryption strength dynamic adaptation formula adjusts the encryption strategy according to the field sensitivity level and real-time security status, ensuring data security while avoiding the performance loss caused by excessive encryption, achieving a balance between security and efficiency. The application of these algorithms eliminates the need for manual intervention in data preprocessing, automatically matching the optimal processing strategy and significantly improving synchronization efficiency and accuracy.
[0016] Front-end orchestration and end-to-end collaboration mechanisms further enhance the system's practicality and manageability. Users can configure synchronization strategies and processes through the front-end interface without needing in-depth knowledge of underlying technical details, lowering the barrier to entry. Configuration information takes effect in real time and permeates all stages of data monitoring, capture, preprocessing, and synchronization, forming a complete closed-loop management system. End-to-end monitoring can track the entire process of data from generation to synchronization completion in real time. Combined with lifecycle management functions, it allows for flexible start, pause, retry, and termination of synchronization tasks, ensuring a stable and reliable synchronization process. The multi-drive collaborative execution mechanism supports multiple synchronization methods and can adapt to the interface specifications of different business systems, greatly expanding the system's applicability. This design makes the entire data synchronization process flexible, controllable, efficient, and intelligent, effectively meeting the data synchronization needs of complex business scenarios. Attached Figure Description
[0017] Figure 1 This is a flowchart of a data synchronization method based on front-end orchestration management. Detailed Implementation
[0018] To make the features and beneficial effects of the present invention more apparent and understandable, the present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0019] Example 1 This embodiment provides a data synchronization method based on front-end orchestration management, such as... Figure 1 As shown, it includes the following steps: By selecting the data table and field information to be monitored through the front-end interface, the monitoring configuration information is generated. The synchronization process logic is arranged according to business needs, and the monitoring configuration information is stored in the configuration center in real time, providing a basic configuration basis for subsequent data monitoring rule generation and preprocessing strategy execution. The data monitoring module periodically retrieves the data table and field information to be monitored from the configuration center and automatically generates real-time monitoring rules to ensure that subsequent data change capture can accurately match the monitoring range of the data table and field information configured by the user. Based on real-time monitoring rules, the data change capture tool monitors the addition, deletion and modification operations of the target database in real time, and encapsulates the change data capture records into standard data units according to a preset format, and synchronously marks the data source information and change type, providing structured data input for message queue distribution; Standard data units are sent to the RocketMQ message queue. The asynchronous buffering mechanism smooths out data spikes and fills in troughs, preventing a sudden increase in pressure on subsequent processing stages due to excessive data volume. Data synchronization continuously pulls standard data units to be processed from the message queue, providing stable data flow support for subsequent multi-dimensional preprocessing. The preprocessing process is initiated by calling the dynamic strategy engine, and the preprocessing log is recorded throughout the process. The preprocessing results are directly pushed to the synchronous driver adaptation module. According to the synchronization process logic, the synchronization driver adaptation module automatically matches the corresponding synchronization driver type, receives the pre-processed standard data unit, completes the secondary adaptation of the data format according to the target business system interface specification, and generates a set of synchronization instructions that can be directly executed by each driver. Based on the synchronization instruction set, various synchronization drivers select to perform data synchronization operations in parallel or serially according to the load and network conditions, and push the processed data to the corresponding business system. During the synchronization process, the driver execution status and time consumption information are recorded in real time, and the execution status information is fed back to the synchronization link monitoring module in real time. Integrate change data capture records, preprocessing logs, and driver execution status to build a full-link monitoring system from data generation to synchronization completion. The monitoring data is synchronized to the lifecycle management module in real time, and automatic alarms for abnormal states are also supported. Based on end-to-end monitoring and combined with lifecycle rules configured on the front end, the system manages the entire lifecycle of synchronization tasks, including starting, pausing, retrying, and terminating them. The management results are updated to the front end configuration interface in real time, allowing users to view and adjust the synchronization process logic on the front end, forming a complete closed-loop management process to ensure that data synchronization tasks can continuously and stably meet business needs.
[0020] Preferably, the data change capture tool includes a data transmission service (DTS) and a data replication service (DRS); the preset format of the standard data unit is JSON.
[0021] Preferably, the fetching frequency of the data to be processed is set according to the preprocessing load. In this embodiment, the default fetching frequency of once every 500 milliseconds is adopted.
[0022] Preferably, the preprocessing procedure is as follows: The dynamic strategy engine pulls data verification rules, field mapping relationship tables and encryption / decryption algorithms from the configuration center. It traverses the data content of standard data units field by field and matches the corresponding data verification rules. For example, it verifies the value range of numeric fields and the format specifications of string fields. Combined with data source information, it automatically enables differentiated verification rules for data from different sources. After the verification is completed, the field mapping stage begins. The dynamic strategy engine associates the original field structure of the current database field information with the field structure of the target database field information based on the mapping relationship table in the configuration center. For standard data units that pass the verification, the engine performs field merging, splitting, or format conversion on the data within the standard data unit based on the mapping relationship table. At the same time, it handles field conflicts according to the priority rules preset by the configuration center. After the standard data unit completes the field mapping, the encryption and decryption process is initiated. According to the sensitive field configuration preset by the configuration center, the sensitive fields of the standard data unit are queried, and the corresponding encryption and decryption algorithm is called to perform encryption and decryption processing on the sensitive fields. The encryption key or decryption certificate is dynamically obtained through the security service.
[0023] Dynamically obtain encryption keys or decryption certificates through security services, specifically: The dynamic policy engine sends a key request to the security service. The request includes field identifiers, algorithm type, and key purpose (encryption or decryption). After receiving a key request, the security service verifies whether the fields have key access permissions. If the verification is successful, the security service generates a one-time key. For cases where only decryption processing is required, the security service calls the decryption certificate of the corresponding algorithm type. Use the key and decryption certificate to perform encryption and decryption operations on the specified fields, and delete the key material after the encryption and decryption process is completed.
[0024] Preferably, the formula for calculating the value range of the numeric field is as follows: ; in, T current This specifies the range of values for the current numeric field. T base The baseline threshold; d This is the fluctuation sensitivity coefficient; s s The standard deviation of historical change data; Sc For business scenario coefficients; Preferred, d Configure according to business tolerance, with a value range of 0.1 to 0.5; Preferred, S c In normal scenarios, the value is 1, and in peak scenarios, it is 1.5.
[0025] Preferably, the field merging is performed by calculating the merging weight of each field based on the field reliability score in the configuration center, the proportion of non-empty fields, and the field business priority. ; in, For the first The fusion weight of each field; This is the balance coefficient; For the first Reliability scores for each field; For the first The percentage of non-empty fields in real time for each field; For the first Business priority of each field; n The total number of fields that need to be merged; The value ranges from 0.3 to 0.7, with the maximum value taken when reliability is prioritized and the minimum value taken when business priority is prioritized. The value range is 0 to 1; The value ranges from 1 to 5, with higher priority values being higher.
[0026] Preferably, the comprehensive encryption strength value S enc The calculation formula is as follows: ; in, L b The baseline strength of the sensitivity level configured for the front end; m This is a risk amplification factor. V s This refers to the real-time security factor of the transmission link. Historical attack risk index; Overall strength of encryption and decryption algorithms S enc Related to key length and number of iterations, S enc =10 corresponds to a 128-bit key length and 1000 iterations. S enc =15 corresponds to a 256-bit key length and 2000 iterations; Preferred, m The value is fixed at 0.2; Preferred, V s The value ranges from 0.8 to 1.2, with lower values indicating less secure links. Preferred, H r The historical attack risk index for sensitive fields over the past 90 days is obtained from security logs. A value of 0 is assigned for fields with no attack records, while for fields with attack records, different historical attack risk indices are mapped to intervals based on the risk score R. H r R < 0.05 H r =0; 0.05≤R<0.30, H r =1.0; 0.30≤R<0.65, H r =1.5; 0.65≤R≤1, H r =2.0; The formula for calculating the risk score R is: ; in, F Frequency factor; S f As the attack source factor; S evf Severity factor; The frequency factor, attack source factor, and severity factor are obtained using normalization, and the calculation method is as follows: ; ; ; in, N Number of attacks; N ref Set a threshold for the number of attacks; S Number of attack sources; S ref Set a preset threshold for the number of attack sources; W score Severity rating; Score ref A threshold is preset for severity scoring; min indicates taking the minimum value; The severity score is determined based on the attack type: port scanning and a single abnormal request are scored 0.5; weak password attempts and abnormal login frequency (≥5 times) are scored 1.0; automated access and abnormal API requests are scored 1.5; injection attempts and RCE attempts are scored 2.0. H r =1.0 corresponds to low-frequency, low-impact attack behaviors; H r =1.5 corresponds to multiple persistent attack behaviors; H r =2.0 corresponds to a clearly malicious and high-risk attack.
[0027] The aforementioned full lifecycle management of synchronous tasks, including starting, pausing, retrying, and terminating, specifically includes: Load lifecycle rule configuration, including the start conditions, pause conditions, maximum number of retries, and execution time window rules for the synchronization task. Before the synchronization task is executed, determine whether to allow the synchronization task to start based on the start conditions. When a timeout, synchronization failure, or link unavailability occurs during the synchronization task, perform a pause or retry operation according to the lifecycle rules. The synchronization task is terminated after the maximum number of retries is reached.
[0028] Example 2 This embodiment provides a data synchronization system based on front-end orchestration management, including: The front-end orchestration and configuration module is used to select the data table and field information to be monitored, generate monitoring configuration information, and store the configuration information in the configuration center in real time. The data monitoring module is used to generate real-time monitoring rules, capture changes in data tables and field information, and encapsulate them into standard data units. The message queue module is used to receive standard data units and call the dynamic strategy engine to preprocess the standard data units. The synchronization adaptation module is used to receive the preprocessing results, generate a synchronization instruction set by matching the corresponding driver type according to the synchronization process logic, and perform data synchronization operations based on the preprocessing results. The end-to-end monitoring module integrates change data capture records, preprocessing logs, and synchronization operation status to generate monitoring data; The lifecycle management module combines monitoring data and front-end configuration information to perform task lifecycle management.
[0029] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A data synchronization method based on front-end orchestration management, characterized in that, Includes the following steps: Select the data table and field information to be monitored through the front-end interface, generate monitoring configuration information, and orchestrate the synchronization process logic according to business needs. The monitoring configuration information is stored in the configuration center in real time, and real-time monitoring rules are generated based on the monitoring configuration information. According to the listening rules, the data table and field information change data are captured, and each change data is encapsulated into a standard data unit and sent to the message queue. The dynamic strategy engine is invoked to pull standard data units from the message queue and preprocess them, and the preprocessing results are pushed to the synchronization adaptation module. The synchronization adaptation module generates a synchronization instruction set by matching the corresponding driver type according to the synchronization process logic, and performs data synchronization operations based on the preprocessing results. The synchronization process is monitored across the entire chain, and task lifecycle management is performed based on the lifecycle rules configured on the front end.
2. The data synchronization method based on front-end orchestration management according to claim 1, characterized in that, The preprocessing specifically includes: The dynamic strategy engine pulls data verification rules, field mapping relationship tables and encryption / decryption algorithms from the configuration center, traverses the data content of standard data units field by field and matches the corresponding data verification rules. The verification rules include verifying the value range of numeric fields and verifying the format specifications of string fields. The field mapping table includes the mapping relationship between the original field structure of the current database field information and the field structure of the target database field information. For standard data units that pass the verification, based on the field mapping table, the data within the standard data units is merged, split, or converted in format. At the same time, field conflicts are handled according to the priority rules preset by the configuration center. For standard data units after field mapping is completed, the sensitive fields are queried according to the sensitive field configuration preset in the configuration center, and the encryption and decryption algorithms are called to perform encryption and decryption processing on the sensitive fields. The encryption key or decryption certificate is dynamically obtained through the security service.
3. The data synchronization method based on front-end orchestration management according to claim 2, characterized in that, The formula for calculating the value range of the numeric field is as follows: ； in, T current This specifies the range of values for the current numeric field. T base The baseline threshold; δ This is the fluctuation sensitivity coefficient; σ s The standard deviation of historical change data; S c For business scenario coefficients.
4. The data synchronization method based on front-end orchestration management according to claim 2, characterized in that, The specific steps for merging the fields are as follows: Based on the field reliability score, the proportion of non-empty fields, and the field business priority in the configuration center, calculate the fusion weight of each field: ； in, For the first The fusion weights of each field; This is the balance coefficient; For the first Reliability scores for each field; For the first The percentage of non-empty fields in each field; For the first Business priority of each field; n This represents the total number of fields that need to be merged.
5. The data synchronization method based on front-end orchestration management according to claim 2, characterized in that, The key length and number of iterations of the encryption / decryption algorithm are set by a preset encryption strength mapping table, which includes the key length and number of iterations corresponding to different encryption strength comprehensive values. The comprehensive value of encryption strength S enc The calculation formula is as follows: ； in, L b The baseline strength for the sensitivity level; μ This is a risk amplification factor. V s This refers to the real-time security factor of the transmission link. This is a historical attack risk index.
6. The data synchronization method based on front-end orchestration management according to claim 1, characterized in that, The changes to the data tables and field information are captured based on data transmission services and data replication services. The default format of the standard data unit is JSON.
7. The data synchronization method based on front-end orchestration management according to claim 1, characterized in that, The preset fetch frequency rules include fetch frequencies under different preprocessing load conditions; Standard data units are retrieved from the message queue according to the retrieval frequency rules.
8. The data synchronization method based on front-end orchestration management according to claim 1, characterized in that, An asynchronous buffering mechanism is used to send standard data units to the message queue.
9. The data synchronization method based on front-end orchestration management according to claim 1, characterized in that, The synchronization process is monitored across the entire link. When an abnormal state is detected, an alarm mechanism is triggered to send the abnormal state information to the user.
10. A data synchronization system based on front-end orchestration management, characterized in that, The system includes: The front-end orchestration and configuration module is used to select the data table and field information to be monitored, generate monitoring configuration information, and store the configuration information in the configuration center in real time. The data monitoring module is used to generate real-time monitoring rules, capture changes in data tables and field information, and encapsulate them into standard data units. The message queue module is used to receive standard data units and call the dynamic strategy engine to preprocess the standard data units. The synchronization adaptation module is used to receive the preprocessing results, generate a synchronization instruction set by matching the corresponding driver type according to the synchronization process logic, and perform data synchronization operations based on the preprocessing results. The end-to-end monitoring module integrates change data capture records, preprocessing logs, and synchronization operation status to generate monitoring data; The lifecycle management module combines monitoring data and front-end configuration information to perform task lifecycle management.