Multi-stage data backflow system and method supporting multiple modes

Abstract

The invention provides a multi-stage data backflow system and method supporting multiple modes. The multi-stage data backflow system comprises an intelligent task scheduling engine, a full-amount synchronization module and an increment synchronization module. The intelligent task scheduling engine is used for determining a data backflow mode based on the data features of the first-level data source and a preset rule; wherein the data backflow mode comprises a full-amount mode and an incremental mode; if the data backflow mode is a full-amount mode, the full-amount synchronization module is used for segmenting full-amount data and synchronizing the segmented data to a second-level node and a third-level node; and if the data backflow mode is an incremental mode, the incremental synchronization module is used for acquiring incremental data based on the database log and the upstream business system change notification, and synchronizing the incremental data to the second-level node and the third-level node. The method can intelligently adapt to different data scenes, and the data synchronization efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of big data synchronization technology, and in particular to a multi-level data backflow system and method that supports multiple modes. Background Technology

[0002] In scenarios such as digital transformation and large enterprise groups, there is a common need to backflow central data to lower-level units according to administrative levels or organizational structures. Existing technical solutions offer two single synchronization methods for handling multi-level data backflow: full update or incremental update. However, they cannot adapt to the differentiated needs of different data volumes (such as small to medium data volumes below one million and large data volumes above one million) and update frequencies (such as low-frequency daily and high-frequency minute-level), making it difficult to balance data timeliness, resource consumption, and operational complexity. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a multi-level data backflow system and method that supports multiple modes, which can intelligently adapt to different data scenarios and improve data synchronization efficiency.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a multi-level data backflow system supporting multiple modes. This system is deployed at least at first-level, second-level, and third-level nodes, and includes: an intelligent task scheduling engine, a full synchronization module, and an incremental synchronization module. The intelligent task scheduling engine is used to determine the data backflow mode based on the data characteristics of the first-level data source and preset rules. The data backflow modes include: a full mode and an incremental mode. If the data backflow mode is a full mode, the full synchronization module is used to segment the full data and synchronize the segmented data to the second-level and third-level nodes. If the data backflow mode is an incremental mode, the incremental synchronization module is used to obtain incremental data based on database logs and upstream business system change notifications, and synchronize the incremental data to the second-level and third-level nodes.

[0005] Optionally, the first-level node includes a first-level push unit, the second-level node includes a second-level relay unit, and the third-level node includes a third-level receiving unit. The intelligent task scheduling engine is specifically used to: collect real-time data characteristics of the first-level data source through data probes, and match the real-time data characteristics with preset rules to determine the data return mode; wherein, the real-time data characteristics include at least: the total amount of data in the data table, the update frequency within a preset time period, and the return frequency of downstream nodes; generate mode instructions based on the data return mode, and synchronously push the mode instructions to the first-level push unit, the second-level relay unit, and the third-level receiving unit.

[0006] Optionally, it also includes: a data sharding rule base, a data sharding execution module, a multi-level state coordinator, and a consistency verification module; the data sharding rule base is used to configure sharding rules for various types of data tables; the data sharding execution module is used to shard data tables based on the sharding rules; the multi-level state coordinator is used to record and manage the data synchronization status of each level node and each data backflow mode, as well as the version information of the sharding rules used. The consistency verification module includes: a pattern verification unit, a data comparison unit, and a difference repair unit. The pattern verification unit is used to verify the data return mode status of each level of node. The data comparison unit is used to compare the total amount of data pushed by the first-level push unit with the total amount of data received by the second-level relay unit and the third-level receiving unit. If there is a difference in the verification, the difference repair unit is triggered to retransmit the missing data.

[0007] Optionally, the full synchronization module includes: a consistent snapshot unit and a data slicing unit; the consistent snapshot unit is used to initiate a snapshot request to the primary data source, obtain full snapshot data at a preset time, and push the full snapshot data to the data slicing execution module, so that the data slicing execution module can determine the first target slicing rule based on the full snapshot data, and perform regional slicing on the full snapshot data based on the first target slicing rule to determine the regional label of each piece of full data; the data slicing unit is used to slice the full data with the same regional label according to the business primary key to obtain sliced ​​data and sliced ​​data fingerprints.

[0008] Optionally, if the data return mode is full mode, the first-level push unit is used to push the fragmented data to the corresponding second-level relay unit according to the regional label of the full data, and synchronize the data synchronization information of the first-level node to the multi-level state coordinator; the second-level relay unit is used to receive the fragmented data of the local region pushed by the first-level push unit, and verify the regional label and fragmented data fingerprint of the fragmented data of the local region. If the verification is successful, the fragmented data of the local region is stored in the local temporary cache, and the receiving status of the second-level node in the multi-level state coordinator is updated; the second-level relay unit is also used to split the fragmented data of the local region based on the third-level regional sub-label to obtain the third-level sub-fragmented data, and push the third-level sub-fragmented data to the corresponding third-level receiving unit, and update the forwarding status of the second-level node in the multi-level state coordinator; the third-level receiving unit is used to receive the third-level sub-fragmented data, and verify the fragmented data fingerprint of the third-level sub-fragmented data. If the verification is successful, the third-level sub-fragmented data is saved to the database, and the receiving result is fed back to the multi-level state coordinator.

[0009] Optionally, the incremental synchronization module includes: a multi-source awareness subunit, an identification subunit, a fault-tolerant processing subunit, and an incremental merging engine; the multi-source awareness subunit is used to monitor the database logs of the primary data source and change notifications from upstream business systems to obtain data change events; the identification subunit is used to parse the data change events to obtain the incremental dataset; wherein, parsing the data change events includes: parsing the database logs to obtain the first changed data; comparing the primary data source table and the incremental cache table based on the business primary key to obtain the second changed data; verifying the version numbers of the first and second changed data and removing duplicates. The system iterates through the data to obtain an incremental dataset. The identification subunit pushes the incremental dataset to the data segmentation execution module, enabling the module to determine a second target segmentation rule based on the incremental dataset. This second target segmentation rule is then used to perform regional segmentation of the incremental data, determining the regional label for each piece of incremental data. The incremental data is then segmented according to a preset grouping size based on the regional labels, resulting in incremental fragment data and corresponding incremental fragment identifiers. The fault-tolerant processing subunit saves the incremental fragment data to a local cache and merges the incremental fragment data using an incremental merging engine to generate a temporary full result set.

[0010] Optionally, if the data return mode is incremental, the first-level push unit is used to push incremental shard data to the corresponding second-level relay unit according to the regional label of the incremental data. If a network interruption occurs during the push process, the breakpoint resume mechanism is triggered to restore data transmission based on the local cache log, and the incremental shard identifier and change timestamp are synchronized to the multi-level state coordinator. The second-level relay unit is used to receive the incremental shard data of the local region pushed by the first-level push unit, and to split the incremental shard data of the local region based on the third-level regional sub-label to obtain the third-level incremental sub-shard data, and to push the third-level incremental sub-shard data to the corresponding third-level receiving unit, and to synchronize the data synchronization information of the second-level node to the multi-level state coordinator. The third-level receiving unit is used to determine the local data table partition to be updated of the third-level node based on the index of the temporary full result set of the third-level incremental sub-shard data, and to merge the third-level incremental sub-shard data into the local full table corresponding to the local data table partition, generate the incremental synchronization result, and synchronize the incremental synchronization result to the multi-level state coordinator.

[0011] Optionally, if the incremental synchronization module experiences anomalies more than a preset number of times, the multi-level state coordinator sends a fault-tolerant trigger signal to the intelligent task scheduling engine to initiate a mode switching process. The mode switching process includes: generating a full-mode temporary instruction and pushing it to the third-level node; triggering the full-synchronization module of the first-level node to generate a full snapshot of the current moment and performing full synchronization; and after the third-level node completes data repair, the intelligent task scheduling engine determines whether to switch to incremental mode based on real-time data characteristics.

[0012] Secondly, the present invention provides a multi-level data backflow method supporting multiple modes, applicable to any of the multi-level data backflow systems supporting multiple modes provided in the first aspect above. This system is deployed at least at first-level nodes, second-level nodes, and third-level nodes, and includes: an intelligent task scheduling engine, a full synchronization module, and an incremental synchronization module. The method includes: determining the data backflow mode based on the data characteristics of the first-level data source and preset rules through the intelligent task scheduling engine; wherein the data backflow mode includes: full mode and incremental mode; if the data backflow mode is full mode, the full synchronization module performs segmentation processing on the full data and synchronizes the segmented data to the second-level and third-level nodes; if the data backflow mode is incremental mode, the incremental synchronization module obtains incremental data based on database logs and upstream business system change notifications, and synchronizes the incremental data to the second-level and third-level nodes.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the steps of the method provided in the second aspect above.

[0014] Fourthly, the present invention provides a computer-readable storage medium on which a computer program is stored, and the computer program, when executed by a processor, performs the steps of the method provided in the second aspect above.

[0015] This invention brings the following beneficial effects: The multi-level data backflow system and method supporting multiple modes provided by this invention are deployed at least at first-level, second-level, and third-level nodes. The system includes: an intelligent task scheduling engine, a full synchronization module, and an incremental synchronization module. The intelligent task scheduling engine determines the data backflow mode based on the data characteristics of the first-level data source and preset rules. The data backflow modes include: full mode and incremental mode. If the data backflow mode is full mode, the full synchronization module segments the full data and synchronizes the segmented data to the second-level and third-level nodes. If the data backflow mode is incremental mode, the incremental synchronization module obtains incremental data based on database logs and upstream business system change notifications and synchronizes the incremental data to the second-level and third-level nodes. This system, through the intelligent task scheduling engine, can match the most efficient data backflow mode according to data characteristics, thereby intelligently adapting to different data scenarios and improving data synchronization efficiency. For incremental mode, it can combine database logs and upstream business system change notifications to obtain incremental data, thereby improving the accuracy of incremental data identification and reducing the operational difficulty of incremental mode.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of a multi-level data backflow system supporting multiple modes, provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of another multi-level data backflow system supporting multiple modes provided in an embodiment of the present invention; Figure 3 A flowchart illustrating a method for supporting multi-level data backflow in both full and incremental modes, as provided in an embodiment of the present invention; Figure 4A flowchart illustrating a method for a multi-level data backflow system supporting multiple modes, provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0020] icon: 10 - Intelligent Task Scheduling Engine; 20 - Full Synchronization Module; 201 - Consistency Snapshot Unit; 202 - Data Slicing Unit; 30 - Incremental Synchronization Module; 301 - Multi-Source Perception Subunit; 302 - Identification Subunit; 303 - Fault-Tolerant Processing Subunit; 304 - Incremental Merging Engine; 40 - Data Segmentation Rule Base; 50 - Data Segmentation Execution Module; 60 - Multi-Level State Coordinator; 70 - Consistency Verification Module; 701 - Pattern Verification Unit; 702 - Data Comparison Unit; 703 - Difference Repair Unit. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Currently, existing technologies in multi-level data backflow not only suffer from problems such as single mode and difficulty in switching, but also fail to address the core technical challenges of the two modes. Specific shortcomings are as follows: (1) The contradiction between mode adaptation and switching is prominent: Most backflow systems only support a single mode and have not designed an adaptation mechanism based on the characteristics of data volume and update frequency. For example, the full mode is used for high-frequency update data of more than one million, which leads to a surge in resource consumption; the incremental mode is used for low-frequency data of less than one million, which results in a waste of operation and maintenance costs. At the same time, mode switching requires manual deployment.

[0023] (2) Difficulty in data segmentation and maintenance: Existing solutions mostly use fixed field segmentation (such as only based on the "regional code" field built into the data table), which cannot adapt to the differentiated segmentation requirements of different tables. For example, household registration data can be directly segmented by the "household registration address" field, but motor vehicle data needs to be associated with an external household registration table to obtain the "vehicle owner's household registration address" before segmentation can be completed. Existing systems lack a unified segmentation rule configuration and external table association mechanism, and independent segmentation scripts need to be developed for each table, resulting in extremely high development and maintenance costs.

[0024] (3) The core difficulties of incremental mode have not been overcome: Incremental backflow faces multiple technical bottlenecks: First, the perception of upstream data changes is lagging, relying on the parsing of a single database log, and the perception of cross-database data changes is not timely; Second, the accuracy of incremental data identification is low, and it is easy to miss field-level updates or mistakenly capture duplicate data; Third, there is a lack of a sound fault tolerance mechanism, and it is impossible to recover quickly after the incremental transmission is interrupted; Fourth, incremental processing and sharing are disconnected, making it difficult to efficiently process scattered incremental data into full results for downstream use; Fifth, the high scheduling frequency leads to the accumulation of job tasks, and the difficulty of operation and maintenance is significantly increased.

[0025] (4) Imbalance in resource management in full mode: Full backflow has a large resource consumption during cross-level synchronization, and existing technologies have not established a dynamic balance mechanism for data volume, backflow frequency and backflow mode, and cannot find a balance between resource consumption and data value.

[0026] (5) Poor consistency in multi-level transfer: When incremental data is transferred from the city level to the county level, there is a lack of mode synchronization mechanism, which may result in a gap of "provincial incremental push → city full storage → county incremental missing"; when full data is transmitted across levels, it is easy to lose fragments due to bandwidth fluctuations, and duplicates or omissions are easy to occur when multi-level node data is merged.

[0027] Based on this, the present invention provides a multi-level data backflow system and method that supports multiple modes, which can intelligently adapt to different data scenarios and improve data synchronization efficiency.

[0028] To facilitate understanding of this embodiment, a multi-level data backflow system supporting multiple modes, as disclosed in this embodiment of the invention, will first be described in detail. This system is deployed at least at first-level nodes, second-level nodes, and third-level nodes; see [link to relevant documentation]. Figure 1 The diagram shows a multi-level data backflow system that supports multiple modes. It illustrates that the system mainly includes: an intelligent task scheduling engine 10, a full synchronization module 20, and an incremental synchronization module 30.

[0029] The intelligent task scheduling engine 10 is used to determine the data return mode based on the data characteristics of the primary data source and preset rules; the data return mode includes: full mode and incremental mode.

[0030] In one implementation, the first-level node includes a first-level push unit, the second-level node includes a second-level relay unit, and the third-level node includes a third-level receiving unit. The intelligent task scheduling engine 10 is specifically used to: collect real-time data characteristics of the first-level data source through a data probe, and match the real-time data characteristics with preset rules to determine the data return mode; wherein, the real-time data characteristics include at least: the total amount of data in the data table, the update frequency within a preset time period, and the return frequency of the downstream node; generate mode instructions based on the data return mode, and synchronously push the mode instructions to the first-level push unit, the second-level relay unit, and the third-level receiving unit.

[0031] In practice, the first-level node can be a provincial-level node, the second-level node can be a municipal-level node, and the third-level node can be a county-level node. In this case, the first-level push unit is a provincial-level push unit, the second-level relay unit is a municipal-level relay unit, and the third-level receiving unit is a county-level receiving unit.

[0032] The intelligent task scheduling engine 10, acting as the system's brain, incorporates built-in policy rules. Based on factors such as data volume, update frequency, data return frequency requirements, and whether it's the first synchronization, it intelligently decides whether to activate full or incremental mode. Specifically, the intelligent task scheduling engine 10 collects real-time data characteristics from provincial data sources via data probes, including the total data volume of the data table, the update frequency over the past month, and the data return frequency requirements of downstream city and county nodes. The intelligent task scheduling engine 10 matches the target mode (i.e., data return mode) according to preset rules. For example, if the collected characteristics meet the criteria of "data volume ≥ 10 million + update frequency ≥ 1 time / hour + data return frequency ≥ 1 time / hour," then it decides to activate incremental mode; if the data volume is < 10 million and the update frequency is 1 time / week, then it decides to activate full mode. The intelligent task scheduling engine generates mode instructions and synchronously pushes these instructions to the provincial push unit, city-level relay unit, and county-level receiving unit, ensuring coordinated operation of the three-level node modes.

[0033] If the data return mode is full mode, the full synchronization module 20 is used to split the full data and synchronize the split data to the secondary and tertiary nodes.

[0034] If the data return mode is incremental mode, the incremental synchronization module 30 is used to obtain incremental data based on database logs and upstream business system change notifications, and synchronize the incremental data to the secondary and tertiary nodes.

[0035] The multi-level data backflow system supporting multiple modes provided in this embodiment of the invention can match the most efficient data backflow mode according to data characteristics through an intelligent task scheduling engine, thereby intelligently adapting to different data scenarios and improving data synchronization efficiency. For incremental mode, incremental data can be obtained by combining database logs and upstream business system change notifications, thereby improving the accuracy of incremental data identification and reducing the operation and maintenance difficulty of incremental mode.

[0036] In one implementation, see Figure 2 As shown, the system also includes: a data segmentation rule base 40, a data segmentation execution module 50, a multi-level state coordinator 60, and a consistency verification module 70.

[0037] The data partitioning rule library 40 is used to configure partitioning rules for various types of data tables. In specific implementations, the data partitioning rule library 40 has pre-defined or supports dynamic configuration of partitioning rules for various types of data tables. For example, for "household registration data", the configuration rule is "partition by household registration address field, and data with a value of 'Hangzhou City' is returned to the Hangzhou node"; for "motor vehicle data", a more complex rule is configured: "associate with the household registration table, using the vehicle owner's ID number as the association key, and obtain their household registration address as the partitioning basis".

[0038] The data segmentation execution module 50 is used to segment the data table based on segmentation rules. In specific implementation, the data segmentation execution module 50 loads the corresponding segmentation rules from the data segmentation rule library 40 according to the instructions of the intelligent task scheduling engine 10 and the needs of the target node, and completes the accurate screening and segmentation of data at the provincial data source (i.e., the first-level data source) to ensure that the data distributed to each node is its own exclusive data.

[0039] The multi-level state coordinator 60 is used to record and manage the data synchronization status of each level node and each data backflow mode, as well as the version information of the slicing rules used. In specific implementation, the multi-level state coordinator 60 is responsible for recording and managing the synchronization status of each level node, each data backflow mode, and the version of the applied slicing rules, enabling breakpoint resumption and global state visibility, and ensuring the consistency of multi-level links.

[0040] The consistency verification module 70 includes: a mode verification unit 701, a data comparison unit 702, and a difference repair unit 703; wherein, the mode verification unit is used to verify the data return mode status of each level of node; the data comparison unit is used to compare the total amount of data pushed by the first-level push unit with the total amount of data received by the second-level relay unit and the third-level receiving unit; if there is a difference in the verification, the difference repair unit is triggered to retransmit the missing data.

[0041] In practical implementation, the mode verification unit 701 of the consistency verification module 70 periodically checks the mode status of the third-level node: if an inconsistency is found between "first-level incremental → second-level full → third-level incremental", it immediately sends a "mode synchronization instruction" to the second-level relay unit to switch the second-level node to incremental mode.

[0042] In full mode, data comparison unit 702 compares the total number of regional fragments pushed by the primary node with the total number received by the secondary / tertiary nodes, and simultaneously samples fragment data to verify its MD5 fingerprint; in incremental mode, it compares the number of regional incremental fragment data entries pushed by the primary node with the number of data entries merged by the secondary / tertiary nodes, and verifies the consistency of the values ​​of the changed fields.

[0043] If a discrepancy is detected during verification, the discrepancy repair unit 703 triggers repair. In full mode, it requests the first-level node to retransmit the missing regional fragments. In incremental mode, it traces back the incremental log of the corresponding region of the first-level node to retransmit the missing data based on the "Last Synchronization Log Sequence Number (LSN)" recorded by the multi-level state coordinator.

[0044] See Figure 2 As shown, the full synchronization module 20 includes a consistency snapshot unit 201 and a data slicing unit 202.

[0045] In full mode, the consistent snapshot unit 201 is used to initiate a snapshot request to the primary data source, obtain full snapshot data at a preset time, and push the full snapshot data to the data splitting execution module 50, so that the data splitting execution module 50 determines the first target splitting rule based on the full snapshot data, and performs regional splitting on the full snapshot data based on the first target splitting rule to determine the regional label of each full data; the data slicing unit 202 is used to slice the full data with the same regional label according to the business primary key to obtain slicing data and slicing data fingerprint.

[0046] In practical implementation, under full-data mode, the consistency snapshot unit 201 initiates a snapshot request to the primary data source to obtain full snapshot data at a certain moment; the data splitting execution module 50 performs regional splitting on the full snapshot data according to preset splitting rules, and assigns a unique regional label to each data item; after completing the regional splitting, the data slicing unit 202 splits the labeled data of the same region into multiple independent shards according to the business primary key and regional label combination strategy, and generates a unique shard identifier and shard data fingerprint for each shard.

[0047] Furthermore, if the data return mode is full mode, the first-level push unit is used to push the fragmented data to the corresponding second-level relay unit according to the regional tags of the full data, and synchronize the data synchronization information of the first-level node to the multi-level state coordinator. In specific implementation, the first-level push unit pushes the fragmented data of the corresponding region in batches to the second-level relay unit according to the regional tags, and at the same time synchronizes the data synchronization information such as "regional tag, number of fragments, and data volume of each fragment" to the multi-level state coordinator.

[0048] The secondary relay unit receives fragmented data from the local domain pushed by the primary push unit and verifies the regional tags and fragment fingerprints of the fragmented data. If the verification passes, the fragmented data is stored in a local temporary cache, and the secondary node's reception status in the multi-level state coordinator is updated. In specific implementation, the secondary relay unit receives fragmented data from the local domain pushed by the primary node. First, it verifies the regional matching of the fragment identifier and the integrity of the fragment fingerprint. If the verification passes, the fragmented data is temporarily stored in a local temporary storage area, and the "secondary reception status" in the multi-level state coordinator is updated to "fragment X received".

[0049] The secondary relay unit is also used to segment the local region's fragmented data based on the tertiary regional sub-label to obtain tertiary sub-fragments, push the tertiary sub-fragments to the corresponding tertiary receiving units, and update the forwarding status of the secondary nodes in the multi-level state coordinator. In specific implementation, the secondary relay unit further segments the local region's fragmented data into tertiary sub-fragments (i.e., county-level sub-fragments) based on the tertiary regional sub-label (i.e., county-level sub-fragments), pushes them to the corresponding tertiary receiving units, and synchronizes the "secondary forwarding status" to the multi-level state coordinator after the push is completed.

[0050] The third-level receiving unit receives third-level sub-fragment data and verifies the fragment data fingerprints of the third-level sub-fragment data. If the verification passes, the third-level sub-fragment data is saved to the database, and the receiving result is fed back to the multi-level state coordinator. In specific implementation, the third-level receiving unit receives the third-level sub-fragment data, verifies that the fingerprints are correct, performs batch storage to the database, and after all sub-fragments have been stored in the database, updates the local status to "full synchronization completed" and feeds back the receiving result to the multi-level state coordinator.

[0051] See Figure 2 As shown, the incremental synchronization module 30 includes: a multi-source sensing subunit 301, an identification subunit 302, a fault-tolerant processing subunit 303, and an incremental merging engine 304.

[0052] The multi-source sensing subunit 301 is used to monitor the database logs of the primary data source and change notifications from upstream business systems to obtain data change events. In specific implementations, the multi-source sensing subunit 301 simultaneously monitors the database logs of the primary data source and change notifications from upstream business systems, capturing data change events in real time.

[0053] The identification subunit 302 is used to parse data change events and obtain incremental datasets. This parsing includes: parsing database logs to obtain the first change data; comparing the primary source table and the incremental cache table based on the business primary key to obtain the second change data; and verifying the version numbers of the first and second change data to remove duplicate change data, thus obtaining the incremental dataset. In practice, the identification subunit 302 performs a "three-engine verification" process on the data change events: ① parsing database logs to extract the first change data; ② comparing the primary source table and the incremental cache table based on the business primary key to obtain the second change data, supplementing any missing change data in the logs; ③ verifying the version numbers of the change data, filtering out duplicate historical changes, and ultimately generating an accurate incremental dataset.

[0054] The identification subunit 302 is also used to push the incremental dataset to the data segmentation execution module, so that the data segmentation execution module can determine the second target segmentation rule based on the incremental dataset, and perform regional segmentation on the incremental data based on the second target segmentation rule, determine the regional label of each incremental data, and segment the incremental data according to the regional label of the incremental data according to the preset group size to obtain incremental fragment data and corresponding incremental fragment identifiers. In specific implementation, after the identification subunit 302 generates an accurate incremental dataset, it pushes it to the data segmentation execution module 50, performs regional segmentation on the incremental data according to the segmentation rule, and assigns a regional label to each incremental data. After completing the regional segmentation, the incremental data is grouped according to the regional label, and each group of data is segmented into incremental fragment data according to the preset size, generating incremental fragment identifiers containing regional information.

[0055] The fault-tolerant processing subunit 303 is used to save incremental sharded data to a local cache and merge the incremental sharded data through the incremental merging engine 304 to generate a temporary full result set. In specific implementation, the fault-tolerant processing subunit 303 caches the incremental sharded data locally, and at the same time merges the incremental data with the existing data according to preset rules (such as "update if primary key matches, insert if no match") through the incremental merging engine to generate a temporary full result set.

[0056] Furthermore, if the data return mode is incremental, the primary push unit pushes incremental fragmented data to the corresponding secondary relay unit according to the geographic label of the incremental data. If a network interruption occurs during the push process, a breakpoint resumption mechanism is triggered to restore data transmission based on local cached logs, and the incremental fragment identifier and change timestamp are synchronized to the multi-level state coordinator. In specific implementation, the primary push unit pushes incremental fragmented data from various regions to the corresponding secondary relay unit in real time. If a network interruption occurs during the push process, a breakpoint resumption mechanism is triggered to restore transmission based on local cached logs, and the incremental identifier and change timestamp are synchronized to the multi-level state coordinator.

[0057] The secondary relay unit receives incremental fragmented data from the local domain pushed by the primary push unit, segments the incremental fragmented data based on the tertiary regional sub-label to obtain tertiary incremental sub-fragments, and pushes the tertiary incremental sub-fragments to the corresponding tertiary receiving units. It also synchronizes the data synchronization information of the secondary nodes to the multi-level state coordinator. In specific implementation, the secondary relay unit receives incremental fragmented data from the local domain pushed by the primary node, automatically synchronizes the provincial incremental mode, segments the incremental fragmented data into tertiary incremental sub-fragments (county-level sub-fragments) based on the tertiary regional sub-label (county-level regional sub-label), and then transmits it to the corresponding tertiary receiving unit. Simultaneously, it records the "secondary transmission time" and "number of incremental data entries" to the multi-level state coordinator. At the secondary node level, the incremental data is merged into the local full table.

[0058] The third-level receiving unit is used to determine the local data table partition to be updated in the third-level node based on the index of the temporary full result set of the third-level incremental sub-shard data, and to merge the third-level incremental sub-shard data into the local full table corresponding to the local data table partition, generating incremental synchronization results, and synchronizing the incremental synchronization results to the multi-level state coordinator. In specific implementation, the third-level receiving unit interfaces with the local incremental merging engine, quickly locates the local data table partition to be updated based on the index of the temporary full result set in the incremental data, executes the "MERGE INTO" operation, and merges the incremental data into the local full table; after completing the merging, the third-level receiving unit generates incremental synchronization results and feeds them back to the multi-level state coordinator.

[0059] In one implementation, if the incremental synchronization module experiences anomalies more than a preset number of times, the multi-level state coordinator sends a fault-tolerant trigger signal to the intelligent task scheduling engine to initiate a mode switching process. This mode switching process includes: generating a full-mode temporary instruction and pushing it to the third-level node; triggering the full-synchronization module of the first-level node to generate a full snapshot of the current moment and performing full synchronization; and after the third-level node completes data repair, the intelligent task scheduling engine determines whether to switch to incremental mode based on real-time data characteristics.

[0060] In practice, if the incremental synchronization module experiences three consecutive anomalies (such as log parsing failure or data loss), the multi-level state coordinator sends a fault-tolerance trigger signal to the intelligent task scheduling engine. The intelligent task scheduling engine initiates a mode switching process, including: generating a temporary instruction for full mode, pushing it to each node, triggering the provincial full synchronization module to generate a consistent snapshot of the current moment (as the incremental repair baseline), and performing a full synchronization. After completing the data baseline repair of the three-level nodes, the intelligent task scheduling engine determines whether to switch back to incremental mode based on real-time data characteristics.

[0061] The system for multi-level data backflow supporting full and incremental modes provided in this embodiment of the invention integrates two core technologies—configurable data splitting rules and full data sharding and incremental log capture (CDC)—by constructing a unified intelligent scheduling framework. It also introduces an automatic mode adaptation and switching mechanism, thereby intelligently adapting to different data scenarios, improving the technical difficulties of incremental mode, and achieving an optimal solution for resource consumption and data timeliness while ensuring the accuracy of localized data distribution.

[0062] For ease of understanding, taking a three-tiered node system (provincial, municipal, and county levels) as an example, this embodiment of the invention also provides a method for implementing multi-level data backflow supporting both full and incremental modes using the aforementioned system. (See [link to relevant documentation]). Figure 3 As shown, it mainly includes the following steps 0 to 12: Step 0: System initialization and parameter configuration.

[0063] Specifically, the system is deployed at provincial, municipal, and county-level nodes, including: initialization and partitioning rule configuration of the intelligent task scheduling engine, full synchronization module, incremental synchronization module, consistency verification module, and multi-level state coordinator; configuring mode decision rules in the intelligent task scheduling engine, and configuring the fault tolerance threshold and incremental data merging rules for incremental mode; initializing the basic state of each level of node in the multi-level state coordinator, recording the first synchronization identifier (e.g., "whether it is the first backflow to the county-level node"), the initial data fingerprint (MD5 value), and the node communication address. Operations personnel use the partitioning rule configuration unit to set partitioning strategies for each backflow table, focusing on configuring the partition size threshold.

[0064] Step 1: Data feature collection and pattern decision.

[0065] The intelligent task scheduling engine collects real-time characteristics of provincial data sources through data probes, including: the total data volume of the data table, the update frequency in the past month, and the return flow frequency requirements of downstream city and county nodes. The intelligent task scheduling engine matches the target mode according to preset rules. For example, if the collected characteristics meet the requirements of "data volume ≥ 10 million + update frequency ≥ 1 time / hour + return flow frequency ≥ 1 time / hour", then the incremental mode is activated; if the data volume is < 10 million and the update frequency is 1 time / week, then the full mode is activated. The intelligent task scheduling engine generates mode instructions and pushes the mode instructions synchronously to the provincial push unit, the city-level relay unit, and the county-level receiving unit to ensure the coordination of the three-level node modes.

[0066] It should be noted that the above preset rules are merely illustrative and can be set according to actual needs, without any restrictions.

[0067] Step 2: Full data processing and provincial push (executed when the decision is full data mode).

[0068] The consistency snapshot unit of the full synchronization module initiates a snapshot request to the provincial data source to obtain a complete data view (i.e., full snapshot data) at a certain time (such as time T0), thus avoiding inconsistencies caused by data changes during the synchronization process.

[0069] The data splitting execution module performs regional splitting on the full snapshot data according to the preset splitting rules, and assigns a unique regional label to each data item (e.g., label 330100 corresponds to Hangzhou, label 330200 corresponds to Ningbo). For complex splitting scenarios that require association with external tables, the full data association is completed first, and then the unified labeling is performed.

[0070] After completing the regional segmentation, the data slicing unit divides the tagged data of the same region into multiple independent slices according to the business primary key and regional label combination strategy (e.g., each city's data is divided into 500,000 records / slice). Each slice generates a unique slice identifier (containing regional information, such as "FULL_330100_001" indicating Hangzhou as the first slice) and a slice data fingerprint.

[0071] The provincial push unit pushes the corresponding regional segmented data to the target city in batches according to the regional tag, and at the same time synchronizes the "regional tag, number of segments, and data volume of each segment" to the multi-level status coordinator.

[0072] Step 3: City-level transfer in full mode (executed when the decision is full mode).

[0073] The municipal relay unit receives fragmented data from the provincial level. First, it verifies the regional matching of the fragment identifier (e.g., the Hangzhou node only receives fragments with the identifier 330100) and the integrity of the fragment data fingerprint. If the verification passes, the fragment data is temporarily stored in the local temporary storage area, and the "municipal-level receiving status" in the multi-level status coordinator is updated to "fragment X received".

[0074] The municipal-level transit unit further divides the local area's fragmented data into county-level sub-fragments based on the county-level regional sub-labels (e.g., 330105 corresponds to Hangzhou's West Lake District), and pushes them to the corresponding county-level receiving unit. After the push is completed, the "municipal-level forwarding status" is synchronized to the multi-level status coordinator.

[0075] Step 4: County-level reception in full mode (executed when the decision is full mode).

[0076] The county-level receiving unit receives the sub-fragment data, verifies the fingerprints, and performs batch data entry. After all sub-fragments have been entered into the database, the local status is updated to "full synchronization completed", and the receiving results are fed back to the multi-level status coordinator.

[0077] Step 5: Incremental data processing and provincial push (executed when the decision is to use incremental mode).

[0078] The multi-source sensing subunit of the incremental synchronization module simultaneously monitors the database logs of the provincial data source (such as MySQL binlog and Oracle redo log) and change notifications from upstream business systems (such as the callback of the insurance information update interface), capturing data change events in real time.

[0079] The identification subunit performs a "three-engine verification" on data change events: ① Parse database logs to extract changed data (such as adding insurance records or updating payment amounts); ② Compare the provincial source table and incremental cache table by business primary key to complete the change data that was missed in the logs; ③ Verify the version number of the changed data, filter out duplicate historical changes, and finally generate an accurate incremental dataset.

[0080] After the incremental synchronization module generates an accurate incremental dataset, it pushes it to the data segmentation execution module, which performs regional segmentation of the incremental data according to the segmentation rules and adds a regional label to each incremental data.

[0081] After completing the regional segmentation, the incremental data is grouped according to the regional label. Each group of data is divided into incremental fragment data according to a preset size (e.g., 10,000 records / fragment), and an incremental fragment identifier containing regional information (e.g., "INC_330100_20240918_001") is generated.

[0082] The fault-tolerant processing subunit caches the incremental sharded data locally (retaining 72 hours of logs), and at the same time merges the incremental data with the existing data according to preset rules (such as "update if primary key matches, insert if no match") through the incremental merging engine to generate a temporary full result set.

[0083] The provincial push unit pushes incremental fragmented data from various regions to the corresponding municipal relay unit in real time. If a network interruption occurs during the push process, the breakpoint resume mechanism is triggered to restore transmission based on local cached logs. At the same time, the incremental identifier and change timestamp are synchronized to the multi-level state coordinator.

[0084] Step 6: Incremental mode city-level transfer (executed when the decision is to use incremental mode).

[0085] The municipal-level transit unit receives incremental fragmented data from the provincial-level push to the local area and automatically synchronizes with the provincial-level incremental mode. After dividing the incremental fragmented data into county-level sub-fragments according to the county-level regional sub-tags, it transmits the data to the corresponding county-level receiving unit. At the same time, it records the "municipal-level transmission time" and "number of incremental data entries" to the multi-level status coordinator. At the municipal level, the incremental data is merged into the local full table.

[0086] Step 7: Incremental mode county-level reception (executed when the decision is in incremental mode).

[0087] The county-level receiving unit connects to the local incremental merging engine. Based on the temporary full result set index in the incremental data, it quickly locates the local data table partition that needs to be updated, executes the "MERGE INTO" operation, and merges the incremental data into the local full table to avoid a full table scan. After the county-level receiving unit completes the merging, it generates an incremental synchronization result (such as "Successfully merged 500 records, no anomalies") and feeds it back to the multi-level status coordinator.

[0088] Step 8: Multi-level consistency check.

[0089] The consistency verification module's mode verification unit periodically checks the mode status of the three-level nodes: if an inconsistency is found between "provincial incremental → municipal full → county incremental", it will immediately send a "mode synchronization instruction" to the municipal relay unit to switch the municipal level to incremental mode.

[0090] The data comparison unit performs verification: In full mode, it compares the total number of regional segments pushed by the provincial level with the total number of segments received by the city / county level, and simultaneously samples segments to verify the MD5 fingerprint; in incremental mode, it compares the number of incremental regional segments pushed by the provincial level with the number of merged data segments by the city / county level, and verifies the consistency of the values ​​of the changed fields.

[0091] If the verification finds a difference (such as the county level receiving one less full fragment than the provincial level, or 10 missing data entries in incremental merging), the difference repair unit triggers repair: In full mode, it requests the provincial level to retransmit the missing regional fragments; in incremental mode, it traces back the incremental log of the corresponding region of the provincial level to retransmit the missing data based on the "Last Synchronization Log Sequence Number (LSN)" recorded by the multi-level state coordinator.

[0092] Step 9: Incremental mode exception handling and mode switching.

[0093] If the incremental synchronization module experiences three consecutive anomalies (such as log parsing failure or data loss), the multi-level state coordinator sends a fault-tolerance trigger signal to the intelligent task scheduling engine. The intelligent task scheduling engine starts the mode switching process as follows: a temporary instruction for full mode is generated and pushed to each node, while triggering the provincial full synchronization module to generate a consistency snapshot of the current moment (as the incremental repair baseline); a full synchronization is performed according to steps 2-3. After the county-level node data baseline is repaired, the intelligent task scheduling engine determines whether to switch back to incremental mode based on real-time data characteristics (if the data update frequency still meets the incremental condition after repair, then switch back to incremental mode).

[0094] Step 10: Synchronize status records and operation and maintenance visualization.

[0095] The multi-level state coordinator updates the synchronization status of each level of nodes in real time, including: mode type, synchronization time, data volume in each region, success / failure indicators, and anomaly repair records, forming a full-process status log. The system synchronizes the status log to the operation and maintenance visualization platform, displaying the distribution of the three-level node modes, the success rate of full / incremental synchronization in each region, and resource utilization (CPU / bandwidth) in chart form. Operation and maintenance personnel can filter and view historical synchronization records and anomaly details by region without the need for manual log investigation.

[0096] Step 11: Periodic pattern optimization.

[0097] The intelligent task scheduling engine periodically performs pattern optimization analysis, such as analyzing the data volume change trend and update frequency fluctuation of various regions based on data from the past week. If the analysis finds that the data characteristics no longer meet the current pattern, the intelligent task scheduling engine automatically adjusts to the full-volume pattern and updates the parameter thresholds in the pattern decision rules. The intelligent task scheduling engine pushes the pattern optimization results and adjustment basis to the operation and maintenance visualization platform for operation and maintenance personnel to confirm (operation and maintenance personnel can manually cancel the adjustment).

[0098] Step 12: Task completion and resource release.

[0099] After a full or incremental synchronization cycle (e.g., 1 day) is completed, the intelligent task scheduling engine sends a task end instruction to each level of node; the full synchronization module releases snapshot resources and deletes temporary fragment files in groups according to region tags; the incremental synchronization module cleans up local cache logs older than 72 hours according to region tags and releases resources.

[0100] The multi-level data backflow system supporting multiple modes provided in the embodiments of the present invention has the following beneficial effects: (1) Intelligent adaptation and doubled efficiency: The system can automatically select the most efficient synchronization mode for data with different characteristics, combining the good consistency of full synchronization with the extremely high timeliness of incremental synchronization, and improving the overall synchronization efficiency by more than 60%.

[0101] (2) The incremental capture mechanism based on CDC fundamentally solves the problem of inaccurate identification of incremental data; through automatic mode switching, a system-level fault tolerance mechanism is provided, reducing the difficulty of operation and maintenance of incremental mode from high-frequency manual intervention to automatic system healing.

[0102] (3) Resource optimization and cost control: It avoids the waste of resources caused by the abuse of full data for small data and the insistence on full data for large data, so that computing and bandwidth resources are used effectively and operating costs are significantly reduced.

[0103] (4) Significantly enhanced reliability: The consistency verification mechanism ensures that even in the most complex incremental synchronization, the system can self-repair, achieving a data reliability of 99.99% and reducing the problem recovery time from hours to minutes.

[0104] In addition to the multi-level data backflow system supporting multiple modes provided in the foregoing embodiments, this invention also provides a multi-level data backflow method supporting multiple modes, see [link to relevant documentation]. Figure 4 The flowchart shown illustrates a multi-level data backflow method supporting multiple modes, indicating that the method mainly includes the following steps S401 to S403: Step S401: Determine the data return mode based on the data characteristics of the primary data source and preset rules through the intelligent task scheduling engine; wherein, the data return mode includes: full mode and incremental mode.

[0105] Step S402: If the data return mode is full mode, the full data is split through the full synchronization module and the split data is synchronized to the secondary and tertiary nodes.

[0106] Step S403: If the data return mode is incremental mode, the incremental synchronization module obtains incremental data based on database logs and upstream business system change notifications, and synchronizes the incremental data to the secondary and tertiary nodes.

[0107] The multi-level data backflow method supporting multiple modes provided in this embodiment of the invention can match the most efficient data backflow mode according to data characteristics through an intelligent task scheduling engine, thereby intelligently adapting to different data scenarios and improving data synchronization efficiency. For incremental mode, incremental data can be obtained by combining database logs and upstream business system change notifications, thereby improving the accuracy of incremental data identification and reducing the operation and maintenance difficulty of incremental mode.

[0108] It should be noted that the method provided in the embodiments of the present invention has the same implementation principle and technical effect as the aforementioned system embodiments. For the sake of brevity, any parts not mentioned in the method embodiments can be referred to the corresponding content in the aforementioned system embodiments.

[0109] This invention also provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0110] Figure 5The present invention provides a schematic diagram of the structure of an electronic device 100, which includes a processor 51, a memory 52, a bus 53 and a communication interface 54. The processor 51, the communication interface 54 and the memory 52 are connected through the bus 53. The processor 51 is used to execute executable modules, such as computer programs, stored in the memory 52.

[0111] The memory 52 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 54 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0112] Bus 53 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0113] The memory 52 is used to store programs. After receiving an execution instruction, the processor 51 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 51 or implemented by the processor 51.

[0114] Processor 51 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 51 or by software instructions. Processor 51 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 52. ​​The processor 51 reads the information in memory 52 and, in conjunction with its hardware, completes the steps of the above method.

[0115] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0116] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-level data backflow system supporting multiple modes, characterized in that, The system is deployed at least on first-level nodes, second-level nodes, and third-level nodes, and includes: an intelligent task scheduling engine, a full synchronization module, and an incremental synchronization module; The intelligent task scheduling engine is used to determine the data return mode based on the data characteristics of the primary data source and preset rules; wherein, the data return mode includes: full mode and incremental mode; If the data return mode is the full mode, the full synchronization module is used to split the full data and synchronize the split data to the second-level node and the third-level node; If the data return mode is the incremental mode, the incremental synchronization module is used to obtain incremental data based on database logs and upstream business system change notifications, and synchronize the incremental data to the secondary node and the tertiary node.

2. The system according to claim 1, characterized in that, The first-level node includes a first-level push unit, the second-level node includes a second-level relay unit, and the third-level node includes a third-level receiving unit; the intelligent task scheduling engine is specifically used for: The real-time data characteristics of the primary data source are collected by a data probe, and the real-time data characteristics are matched with preset rules to determine the data return mode; wherein, the real-time data characteristics include at least: the total amount of data in the data table, the update frequency within a preset time period, and the return frequency of the downstream node; Based on the data return mode, a mode instruction is generated and the mode instruction is synchronously pushed to the first-level push unit, the second-level relay unit, and the third-level receiving unit.

3. The system according to claim 2, characterized in that, Also includes: Data partitioning rule base, data partitioning execution module, multi-level state coordinator, and consistency verification module; The data partitioning rule library is used to configure partitioning rules for various types of data tables; The data splitting execution module is used to split the data table based on the splitting rules; The multi-level state coordinator is used to record and manage the data synchronization status of each level node and each data backflow mode, as well as the version information of the slicing rules adopted. The consistency verification module includes: a mode verification unit, a data comparison unit, and a difference repair unit; wherein, the mode verification unit is used to verify the data backflow mode status of each level of node; the data comparison unit is used to compare the total amount of data pushed by the first-level push unit with the total amount of data received by the second-level relay unit and the third-level receiving unit; if there is a difference in the verification, the difference repair unit is triggered to retransmit the missing data.

4. The system according to claim 3, characterized in that, The full synchronization module includes: a consistent snapshot unit and a data slicing unit; The consistency snapshot unit is used to initiate a snapshot request to the primary data source, obtain full snapshot data at a preset time, and push the full snapshot data to the data segmentation execution module, so that the data segmentation execution module determines a first target segmentation rule based on the full snapshot data, and performs regional segmentation on the full snapshot data based on the first target segmentation rule to determine the regional label of each piece of full data. The data slicing unit is used to slice the full data of the same geographic label according to the business primary key to obtain slicing data and slicing data fingerprints.

5. The system according to claim 4, characterized in that, If the data return mode is full mode, the first-level push unit is used to push the fragmented data to the second-level relay unit of the corresponding region according to the regional label of the full data, and synchronize the data synchronization information of the first-level node to the multi-level state coordinator. The secondary relay unit is used to receive the local domain fragment data pushed by the primary push unit, and to verify the regional label and fragment data fingerprint of the local domain fragment data. If the verification is successful, the local domain fragment data is stored in the local temporary cache area, and the receiving status of the secondary node in the multi-level state coordinator is updated. The secondary relay unit is also used to divide the fragmented data of the local domain based on the third-level regional sub-label to obtain third-level sub-fragmented data, push the third-level sub-fragmented data to the corresponding third-level receiving unit, and update the forwarding status of the secondary nodes in the multi-level state coordinator. The third-level receiving unit is used to receive the third-level sub-fragment data, verify the fragment data fingerprint of the third-level sub-fragment data, and if the verification is successful, save the third-level sub-fragment data to the database and feed back the receiving result to the multi-level state coordinator.

6. The system according to claim 3, characterized in that, The incremental synchronization module includes: a multi-source sensing subunit, an identification subunit, a fault-tolerant processing subunit, and an incremental merging engine; The multi-source sensing subunit is used to monitor the database logs of the primary data source and change notifications from the upstream business system to obtain data change events; The identification subunit is used to parse the data change event and obtain an incremental dataset; wherein, parsing the data change event includes: parsing the database log to obtain first changed data; comparing the primary data source table and the incremental cache table based on the business primary key to obtain second changed data; verifying the version numbers of the first changed data and the second changed data, removing duplicate changed data, and obtaining the incremental dataset; The identification subunit is further configured to push the incremental dataset to the data segmentation execution module, so that the data segmentation execution module determines a second target segmentation rule based on the incremental dataset, performs regional segmentation on the incremental data based on the second target segmentation rule, determines the regional label of each incremental data, and segments the incremental data according to the regional label of the incremental data according to a preset group size to obtain incremental fragment data and corresponding incremental fragment identifiers; The fault-tolerant processing subunit is used to save the incremental shard data to the local cache and merge the incremental shard data through the incremental merging engine to generate a temporary full result set.

7. The system according to claim 6, characterized in that, If the data return mode is the incremental mode, the first-level push unit is used to push the incremental shard data to the corresponding second-level relay unit according to the regional label of the incremental data; If a network interruption occurs during the push process, the breakpoint resume mechanism is triggered to restore data transmission based on the local cache logs, and the incremental fragment identifier and change timestamp are synchronized to the multi-level state coordinator. The secondary relay unit is used to receive incremental fragmented data of the local domain pushed by the primary push unit, and to divide the incremental fragmented data of the local domain based on the third-level regional sub-label to obtain third-level incremental sub-fragmented data, and to push the third-level incremental sub-fragmented data to the corresponding third-level receiving unit, and to synchronize the data synchronization information of the secondary node to the multi-level state coordinator. The third-level receiving unit is used to determine the local data table partition to be updated of the third-level node based on the index of the temporary full result set of the third-level incremental sub-shard data, and to merge the third-level incremental sub-shard data into the local full table corresponding to the local data table partition, generate an incremental synchronization result, and synchronize the incremental synchronization result to the multi-level state coordinator.

8. The system according to claim 3, characterized in that, If the incremental synchronization module experiences an anomaly more than a preset number of times, the multi-level state coordinator sends a fault-tolerant trigger signal to the intelligent task scheduling engine, causing the intelligent task scheduling engine to initiate a mode switching process. This mode switching process includes: generating a temporary full-mode instruction and pushing it to the third-level node; triggering the full-synchronization module of the first-level node to generate a full snapshot of the current moment and performing full synchronization; and after the third-level node completes data repair, the intelligent task scheduling engine determines whether to switch to incremental mode based on real-time data characteristics.

9. A multi-level data backflow method supporting multiple modes, characterized in that, The method is applied to the multi-level data backflow system supporting multiple modes as described in any one of claims 1 to 8, wherein the system is deployed at least at first-level nodes, second-level nodes, and third-level nodes, and includes: an intelligent task scheduling engine, a full synchronization module, and an incremental synchronization module; the method includes: The intelligent task scheduling engine determines the data return mode based on the data characteristics of the primary data source and preset rules; wherein, the data return mode includes: full mode and incremental mode; If the data return mode is the full mode, the full data is segmented by the full synchronization module, and the segmented data is synchronized to the second-level node and the third-level node. If the data return mode is the incremental mode, the incremental synchronization module obtains incremental data based on database logs and upstream business system change notifications, and synchronizes the incremental data to the secondary node and the tertiary node.

10. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the steps of the method of claim 9.

Images