Incremental data synchronization method, apparatus and storage medium

Abstract

This disclosure proposes an incremental data synchronization method, apparatus, and storage medium, comprising: determining multiple first update times corresponding to multiple data nodes based on update time attributes carried during registration, comparing the multiple first update times with multiple second update times respectively to identify target data nodes with time differences, wherein the second update time is the time when the data node last performed a data update, and creating subtasks according to the number of target data nodes; and using the subtasks to obtain incremental data from the corresponding target data nodes and synchronizing the incremental data to the target storage location, which can accurately identify target data nodes among multiple registered data nodes that have undergone incremental changes, and create subtasks for each target data node to synchronize incremental data, thereby improving data synchronization efficiency and saving computing resources.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to an incremental data synchronization method, apparatus, and storage medium. Background Technology

[0002] Data Oriented Architecture (DOA), built upon cloud computing hardware architecture, adopts the concept of "data-oriented and data-centric." It unifies the definition, management, and provision of data services through a Data Registry Center (DRC), a Data Authority Center (DAC), and a Data Exception Control Center (DEC). Data Application Units (DAUs) manage and serve various applications, establishing a data ecosystem of a large data platform and fragmented applications. It constructs a complete mechanism from data protection to application authorization, providing fundamental theoretical and methodological support for effectively addressing the software architecture challenges faced in the era of big data.

[0003] ETL is an abbreviation for Extract-Transform-Load, used to describe the process of extracting, transforming, and loading data from a source to a destination.

[0004] In related technologies, ETL is based on extracting and aggregating specific data. However, within the DOA (Data on Application) architecture, because data is registered separately by each data node and stored locally in a distributed manner, there is no data centralization. This means that incremental data synchronization requires extracting incremental data from all data nodes, making it impossible to synchronize incremental data across all nodes in a single task. Furthermore, different data nodes may use heterogeneous data sources, such as databases like MySQL and Oracle, making incremental data synchronization for heterogeneous data impossible. Summary of the Invention

[0005] This application proposes an incremental data synchronization method, apparatus, and storage medium, aiming to at least partially solve one of the technical problems in the related art.

[0006] The first aspect of this application proposes an incremental data synchronization method, comprising: determining multiple first update times corresponding to multiple data nodes based on update time attributes carried during registration; comparing the multiple first update times with multiple second update times respectively to determine target data nodes with time differences, wherein the second update time is the time when the data node last performed a data update; creating subtasks based on the number of target data nodes; and using the subtasks to obtain incremental data from the corresponding target data nodes and synchronizing the incremental data to the target storage location.

[0007] A second aspect of this application provides an incremental data synchronization device, comprising: a first determining module, configured to determine multiple first update times corresponding to multiple data nodes based on update time attributes carried during registration; a second determining module, configured to compare the multiple first update times with multiple second update times respectively to determine target data nodes with time differences, wherein the second update time is the time when the data node last performed a data update; a task creation module, configured to create subtasks based on the number of target data nodes; and a data synchronization module, configured to use the subtasks to obtain incremental data from the corresponding target data nodes and synchronize the incremental data to the target storage location.

[0008] A third aspect of this application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the incremental data synchronization method of the embodiments of this application.

[0009] A fourth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the incremental data synchronization method disclosed in the embodiments of this application.

[0010] In this embodiment, multiple first update times corresponding to multiple data nodes are determined based on the update time attributes carried during the registration of multiple data nodes. These first update times are then compared with multiple second update times to identify target data nodes with time differences. The second update time is the time when the data node last updated. Subtasks are created based on the number of target data nodes. Incremental data is obtained from the corresponding target data nodes using these subtasks, and the incremental data is synchronized to the target storage location. This approach can accurately identify target data nodes among multiple registered data nodes that have undergone incremental changes. Subtasks are created for each target data node to synchronize incremental data, improving data synchronization efficiency and saving computing resources.

[0011] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a flowchart illustrating an incremental data synchronization method according to an embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram of the incremental data synchronization process provided according to an embodiment of this disclosure;

[0015] Figure 3 This is a flowchart illustrating an incremental data synchronization method according to another embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram of an incremental data synchronization device according to another embodiment of the present disclosure;

[0017] Figure 5 A block diagram of an exemplary computer device suitable for implementing embodiments of the present application is shown. Detailed Implementation

[0018] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0019] In response to the problem mentioned in the background art that in the DOA system, since data is registered separately by each data node and stored locally in a distributed manner without data centralization, incremental data synchronization requires extraction from all data nodes, making it impossible to synchronize incremental data in a single task, this embodiment provides an incremental data synchronization method. The method will be described below with reference to specific embodiments.

[0020] It should be noted that the executing entity of the incremental data synchronization method in this embodiment can be an incremental data synchronization device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0021] Figure 1This is a flowchart illustrating an incremental data synchronization method according to an embodiment of the present disclosure, as shown below. Figure 1 As shown, the method includes:

[0022] S101: Determine multiple first update times corresponding to multiple data nodes based on the update time attribute carried during the registration of multiple data nodes.

[0023] In this embodiment of the disclosure, the idea of ​​"data-oriented and data-centric" can be adopted to manage the data of multiple data nodes, and the multiple data nodes can be registered on the data platform, for example, they can be registered in the DOA system.

[0024] Among them, multiple data nodes can be data nodes from any field, such as governments, companies, banks, etc. There are no restrictions on this.

[0025] For example, Figure 2 This is a schematic diagram of the incremental data synchronization process provided according to an embodiment of this disclosure, such as... Figure 2 As shown, multiple data nodes, such as Level 1 data node 1, Level 2 data node 2-1, Level 2 data node 2-2, Level 3 data node 3-1, Level 3 data node 3-2, etc., can register with the DOA registry center.

[0026] In some embodiments, the multiple data nodes can be in a distributed structure, and the multiple data nodes can be divided into first-level data nodes and lower-level data nodes, such as... Figure 2 As shown, for example, Level 1 data node 1, Level 2 data node 2-1, and Level 3 data node 3-1 can be considered as Level 1 data nodes, while Level 2 data node 2-2 can be a subordinate data node of Level 2 data node 2-1. Similarly, Level 3 data node 3-2 is a subordinate data node of Level 3 data node 3-1. It is understood that in this embodiment, there can be one or more Level 1 data nodes, and each Level 1 data node can include multiple subordinate data nodes; this is not limited.

[0027] Therefore, the incremental data synchronization method provided in this embodiment can be applied to distributed data nodes, enabling the synchronization of incremental data on distributed data nodes.

[0028] In this embodiment of the present disclosure, multiple data nodes can carry an update time attribute during the registration process with the DOA registry center. When the data of multiple data nodes is updated (e.g., data addition, change, etc.), they can report the data update time to the DOA registry center. The DOA registry center can synchronously refresh the update time attribute according to the reported update time. The time under this data time attribute can be called the first update time. That is to say, the first update time is the time when the data of multiple data nodes changes this time.

[0029] In some embodiments, the first update time can be determined periodically by the ETL tool (e.g., daily, hourly, etc.). That is to say, in practical applications, the incremental data synchronization method provided in this disclosure can be executed periodically by the ETL tool.

[0030] Understandably, data nodes can carry attributes such as category code and category name during registration, in addition to update time. There are no restrictions on this. The category code is used to record hierarchical relationships between nodes; for example, the category code could be a code between upper and lower level governments, which can serve as a means of implementing subsequent query and access paths.

[0031] S102: Compare multiple first update times with multiple second update times to determine the target data nodes with time differences.

[0032] Among them, the first update time is the current data update time of the data node, and the second update time is the last data update time of the data node.

[0033] In some embodiments, before determining the multiple first update times corresponding to multiple data nodes, the ETL tool may first determine the access paths of the first-level data nodes and the access paths of the lower-level data nodes.

[0034] The path used to access data nodes can be referred to as the access path (or access address). During the registration process, a first-level data node accesses the DOA data registry. In this case, the ETL tool can determine the access path of the first-level data node. Since the lower-level data nodes and the first-level data nodes are in a distributed structure, a first-level data node can have access paths for its lower-level data nodes. Therefore, the ETL tool can obtain the access paths of lower-level data nodes from the first-level data nodes. In other embodiments, the ETL tool can also query the DOA data registry for the access paths of lower-level data nodes based on category codes. For example, based on the hierarchical relationship recorded in the category codes, it can determine the lower-level data nodes corresponding to the upper-level data node (i.e., the first-level data node) and thus determine the access path. There are no restrictions on this.

[0035] Furthermore, the ETL tool retrieves data attribute information from multiple data nodes based on the access paths of the first-level and lower-level data nodes. That is, the ETL tool queries multiple data nodes for the corresponding data table's data attributes based on the access paths. Data attributes include, for example, table name, data source name, data source type, data source address, access permissions, and source data update time (i.e., the second update time), without any restrictions. Further, a task table is constructed based on the data attribute information; that is, the data attributes of each data node are summarized in the ETL tool's task table. The structure of this task table can be shown in Table 1.

[0036] Table 1

[0037]

[0038] Each data node can correspond to a synchronization task, a data table name, a data source name (i.e., a data node name), a data source address, a data source type (i.e., a database type), access permissions, a local data update time (corresponding to the first update time), a source data update time (corresponding to the second update time), and a record position (used to record the end position of the data update).

[0039] In other embodiments, data nodes may also carry the aforementioned data attributes during the registration process. In this case, the ETL tool can directly construct the task table based on the carried data attributes. The construction process and structure of the task table are not limited here.

[0040] In this embodiment, for example, multiple second update times for multiple data nodes can be obtained based on the source data update time field in the task table. The multiple first update times are then compared with the multiple second update times to determine the target data node with a time difference. For example, if the first update time and second update time of level 2 node 2-1 are different, then level 2 node 2-1 can be considered the target data node. In other words, if the current update time differs from the previous update time, it indicates that the data of that node has changed, and therefore, that node needs to perform data synchronization.

[0041] S103: Create a subtask based on the number of target data nodes.

[0042] After identifying the target data nodes, the ETL tool further creates subtasks based on the number of target data nodes. That is, it creates corresponding subtasks for each target data node (i.e., the synchronization tasks in Table 1).

[0043] S104: Use a subtask to obtain incremental data from the corresponding target data node and synchronize the incremental data to the target storage location.

[0044] Furthermore, such as Figure 2 As shown, the ETL tool uses each subtask to obtain incremental data from the corresponding target data node and synchronizes the incremental data to the target storage location, that is, it synchronizes the incremental data of each target data node to the destination.

[0045] It is understood that the above is only an illustrative example of incremental data synchronization. In practical applications, it is also possible to synchronize changed data of target data nodes and other types of changed data, without any restrictions.

[0046] In some embodiments, after the incremental data is synchronized to the target storage location, the corresponding subtasks can be released and destroyed, thereby saving computing resources.

[0047] In this embodiment, multiple first update times corresponding to multiple data nodes are determined based on the update time attributes carried during the registration of multiple data nodes. These first update times are then compared with multiple second update times to identify target data nodes with time differences. The second update time is the time when the data node last updated. Subtasks are created based on the number of target data nodes. Incremental data is obtained from the corresponding target data nodes using these subtasks, and the incremental data is synchronized to the target storage location. This approach can accurately identify target data nodes among multiple registered data nodes that have undergone incremental changes. Subtasks are created for each target data node to synchronize incremental data, improving data synchronization efficiency and saving computing resources.

[0048] Figure 3 This is a flowchart illustrating an incremental data synchronization method according to another embodiment of the present disclosure, such as... Figure 3 As shown, the method includes:

[0049] S301: Determine multiple first update times corresponding to multiple data nodes based on the update time attribute carried during the registration of multiple data nodes.

[0050] S302: Compare multiple first update times with multiple second update times to determine the target data nodes with time differences.

[0051] S303: Create a subtask based on the number of target data nodes.

[0052] For detailed descriptions of S301 to S303, please refer to the above embodiments, which will not be repeated here.

[0053] S304: Determine the data source type of the target data node based on the task table.

[0054] In this embodiment of the disclosure, the data attribute information may include a data source type attribute, wherein the data source type is used to describe the storage type of the database in the data node. The data source type is, for example, MySQL, Oracle, and any other possible type, without limitation. That is to say, multiple data nodes may have different data source types.

[0055] In this case, in the process of obtaining incremental data from the corresponding target data node using subtasks in the embodiments of this disclosure, the data source type of the target data node can first be determined according to the task table.

[0056] For example, if the target data nodes are "Level 2 Node 2-1" and "Level 2 Node 2-2", this embodiment of the disclosure obtains the data source type of "Level 2 Node 2-1" as MySQL and "Level 2 Node 2-2" as Oracle by querying the data source type field in the task table.

[0057] S305: Determine the target processing method based on the data source type.

[0058] Furthermore, the target processing method is determined based on the data source type; that is, different data source types can correspond to different target processing methods.

[0059] The processing methods include, for example, methods for acquiring incremental data and methods for parsing logs, and there are no restrictions on these methods.

[0060] For example, if the data source type for "Level 2 Node 2-1" is MySQL, then the incremental data acquisition method is to obtain binlog logs, and the corresponding log parsing method is to read the binlog logs and parse out the INSET, UPDATE, and DELETE SQL operations. Conversely, if the data source type for "Level 2 Node 2-2" is Oracle, then the incremental data acquisition method is to obtain redo log logs and use the corresponding Oracle log parsing method.

[0061] S306: Use subtasks to obtain incremental data in a target-oriented manner.

[0062] Furthermore, subtasks are used to acquire incremental data from the corresponding target data nodes in a target processing manner, and the incremental data is synchronized to the target storage location.

[0063] During the synchronization process, each subtask can be converted into a consistent SQL statement and data format based on the storage type of the target storage location, and then written to the target storage location.

[0064] Therefore, the embodiments of this disclosure can create different data synchronization methods for heterogeneous data sources in a single task, thus adapting to different application scenarios and improving the synchronization capability of incremental data.

[0065] In some embodiments, after incremental data synchronization is completed, the status of the task table can also be updated synchronously.

[0066] Specifically, the second update time of the corresponding target data node in the task table can be replaced with the first update time of the target data node. In other words, the local data update time of the target data node in the task table is synchronized to the determined source data update time. Furthermore, the end position of the incremental data for the target data node can be recorded in the record position field of the task table.

[0067] This helps to accurately determine whether incremental data changes have occurred in data nodes and to accurately locate the starting position of incremental data.

[0068] In other embodiments, in practical applications, multiple data nodes may be added or modified. In this case, embodiments of this disclosure can also periodically or manually trigger the ETL tool to query the first-level data node to obtain the access path of the corresponding data table on its subordinate data nodes, thereby determining whether a new data node has appeared based on the path. Furthermore, if a new data node appears, the access path of the new data node can be obtained, and the data attributes of the new data node can be obtained based on the access path and updated to the task table.

[0069] In the case of a newly added data node being a first-level data node, the access path can be obtained from the registration process; in the case of a newly added data node being a subordinate data node, the access path can be obtained from the first-level data node to which the newly added data node belongs.

[0070] Therefore, the embodiments of this disclosure can update the task table to ensure that the data nodes in the task table correspond to the actual application scenario in real time, thereby improving the system's adaptability.

[0071] In this embodiment, multiple first update times corresponding to multiple data nodes are determined based on the update time attributes carried during registration. These first update times are then compared with multiple second update times to identify target data nodes with time differences. The second update time is the time when the data node last updated. Subtasks are created based on the number of target data nodes. These subtasks retrieve incremental data from the corresponding target data nodes and synchronize the incremental data to the target storage location. This accurately identifies target data nodes among the registered data nodes that have undergone incremental changes. Creating subtasks for each target data node to synchronize incremental data improves data synchronization efficiency and saves computing resources. Furthermore, it facilitates accurate subsequent determination of whether data nodes have experienced incremental data changes and accurately locates the starting position of the incremental data. Moreover, this embodiment allows for updating the task table to ensure that the data nodes in the task table correspond to the actual application scenario in real time, improving the system's adaptability.

[0072] Figure 4 This is a schematic diagram of an incremental data synchronization device according to another embodiment of the present disclosure. Figure 4 As shown, the incremental data synchronization device 40 includes:

[0073] The first determining module 401 is used to determine multiple first update times corresponding to multiple data nodes based on the update time attribute carried by multiple data nodes during registration.

[0074] The second determining module 402 is used to compare multiple first update times with multiple second update times respectively to determine the target data node with time difference, wherein the second update time is the time when the data node last had a data update.

[0075] Task creation module 403 is used to create subtasks based on the number of target data nodes; and

[0076] The data synchronization module 404 is used to obtain incremental data from the corresponding target data node using subtasks and synchronize the incremental data to the target storage location.

[0077] In some embodiments, the multiple data nodes include a primary data node and subordinate data nodes of the primary data node.

[0078] In some embodiments, the device 40 further includes: a third determining module, configured to determine the access path of the primary data node and obtain the access path of the secondary data node from the primary data node; and a building module, configured to obtain data attribute information from multiple data nodes based on the access path of the primary data node and the access path of the secondary data node, and build a task table based on the data attribute information.

[0079] In some embodiments, the second determining module 402 includes: a first determining submodule, configured to determine a plurality of second update times according to a task table; and a comparison submodule, configured to compare the plurality of first update times with the plurality of second update times respectively.

[0080] In some embodiments, the data attribute information includes a data source type attribute. The data synchronization module 403 includes: a second determining submodule, used to determine the data source type of the target data node according to the task table; a third determining submodule, used to determine the target processing method according to the data source type; and an acquisition submodule, used to acquire incremental data using subtasks in the target processing method.

[0081] In some embodiments, the apparatus 40 further includes: a first update module, configured to replace the second update time of the corresponding target data node in the task table with the first update time of the target data node; and a second update module, configured to record the end position of the incremental data of the target data node in the task table.

[0082] In some embodiments, the device 40 further includes: a determination module for determining whether a new data node has appeared; and a third update module for obtaining the data attributes of the new data node and updating the task table when a new data node appears.

[0083] In some embodiments, device 40 further includes a release module for releasing the subtask.

[0084] In this embodiment, multiple first update times corresponding to multiple data nodes are determined based on the update time attributes carried during the registration of multiple data nodes. These first update times are then compared with multiple second update times to identify target data nodes with time differences. The second update time is the time when the data node last updated. Subtasks are created based on the number of target data nodes. Incremental data is obtained from the corresponding target data nodes using these subtasks, and the incremental data is synchronized to the target storage location. This approach can accurately identify target data nodes among multiple registered data nodes that have undergone incremental changes. Subtasks are created for each target data node to synchronize incremental data, improving data synchronization efficiency and saving computing resources.

[0085] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0086] To implement the above embodiments, this application also proposes a computer program product that, when the instruction processor in the computer program product is executed, performs the incremental data synchronization method proposed in the foregoing embodiments of this application.

[0087] Figure 5A block diagram of an exemplary computer device suitable for implementing embodiments of the present application is shown. Figure 5 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0088] like Figure 5 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0089] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0090] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0091] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive".

[0092] although Figure 5Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0093] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.

[0094] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 12, and / or with any device that enables computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0095] The processing unit 16 executes various functional applications by running programs stored in the system memory 28, such as implementing the incremental data synchronization method mentioned in the foregoing embodiments.

[0096] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0097] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0098] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0099] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0100] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0101] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0103] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0105] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An incremental data synchronization method, characterized in that, include: Determine the access path of the primary data node, and obtain the access path of the secondary data node from the primary data node; Based on the access paths of the first-level data nodes and the access paths of the lower-level data nodes, data attribute information is obtained from multiple data nodes, and a task table is constructed based on the data attribute information, wherein the data attribute information includes data source type attributes. Based on the update time attribute carried during the registration of the multiple data nodes, multiple first update times corresponding to the multiple data nodes are determined. The multiple data nodes are nodes of a data-oriented architecture (DOA) system. The multiple data nodes include first-level data nodes and lower-level data nodes of the first-level data nodes. The plurality of first update times are compared with the plurality of second update times to determine the target data nodes with time differences, wherein the second update time is the time when the data node last had a data update; Create subtasks based on the number of target data nodes; and The data source type of the target data node is determined based on the task table; Determine the target processing method based on the data source type; The subtask is used to acquire the incremental data in the target processing method, and the incremental data is synchronized to the target storage location.

2. The method as described in claim 1, characterized in that, Comparing the plurality of first update times with the plurality of second update times, including: The plurality of second update times are determined according to the task table; and The plurality of first update times are compared with the plurality of second update times respectively.

3. The method as described in claim 1, characterized in that, After synchronizing the incremental data to the target storage location, the process also includes: The second update time of the corresponding target data node in the task table is replaced with the first update time of the target data node; and The task table records the end position of the incremental data for the target data node.

4. The method as described in claim 1, characterized in that, Also includes: Determine if any new data nodes have been added; as well as When a new data node is added, the data attributes of the new data node are obtained and updated in the task table.

5. The method as described in claim 1, characterized in that, After synchronizing the incremental data to the target storage location, the process also includes: releasing the subtask.

6. An incremental data synchronization device, characterized in that, The apparatus is used to implement the incremental data synchronization method as described in claim 1, the apparatus comprising: The first determining module is used to determine multiple first update times corresponding to the multiple data nodes based on the update time attribute carried by the multiple data nodes during registration; The second determining module is used to compare the plurality of first update times with the plurality of second update times respectively to determine the target data node with time difference, wherein the second update time is the time when the data node last had a data update; The task creation module is used to create subtasks based on the number of target data nodes; and The data synchronization module is used to obtain incremental data from the corresponding target data node using the subtask, and synchronize the incremental data to the target storage location.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

Images