Incremental data offline synchronization method in data development

Through the offline synchronization method of incremental data, whether to call the incremental synchronization tool is determined based on the amount of business data, the operation pressure and data real-time problems caused by the unified batch synchronization tool are solved, and the flexible synchronization and security of data are achieved.

CN118964489APending Publication Date: 2024-11-15HANGZHOU GUANSHU INFORMATION TECH CO LTD (CHINA)
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411110714.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-12-15
Filing Date
2024-08-14
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

During the data development process, a unified offline and batch synchronization tool causes multiple people to call it online at the same time, occupying memory and processes, causing operational pressure, and being unable to follow the changes in business data volume, affecting the real-time storage and security of data.

Method used

Using the offline synchronization method of incremental data, by writing synchronization program scripts and configuring data incremental synchronization tools, determine whether to call incremental synchronization tools based on the data volume of the business data set, perform real-time streaming synchronization, avoid frequent use of fixed tools, and reduce memory Occupy and operating pressure.

Benefits of technology

It realizes flexible call to incremental synchronization tools based on the amount of business data, reduces the pressure of background operation, and improves system security and real-time and consistency of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964489A_ABST
    Figure CN118964489A_ABST
Patent Text Reader

Abstract

The invention relates to an incremental data offline synchronization method in data development, and the method comprises the steps: carrying out the threshold comparison according to the business table data size of each developer, judging whether to call a preset data incremental synchronization tool (Maxwell or Cana), if yes, calling the data incremental synchronization tool, and otherwise, carrying out the data synchronization through a written script. Therefore, the increment synchronization tool is flexibly called according to the numerical value of the service data set, and a fixed increment synchronization tool can be prevented from being frequently scheduled and used, so that the situation that the increment synchronization tool frequently occupies a background running memory is reduced, the data running pressure is reduced, and the safety of a background running system is improved. And meanwhile, incremental synchronization and updating of the data can be realized, and the consistency and the real-time performance of the data are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data set technology, and in particular to an incremental data offline synchronization method, system and electronic device in data development. Background Art

[0002] There are two main ways to synchronize data offline: full synchronization and incremental synchronization.

[0003] Full synchronization is to synchronize all the data in the business database to the data warehouse every day. This is the simplest way to ensure data synchronization on both sides. For tables using full synchronization, full synchronization is usually required on the first day.

[0004] Incremental synchronization is to synchronize only the new and changed data in the business data to the data warehouse every day. For tables that use daily incremental synchronization, it is usually necessary to perform a full synchronization on the first day.

[0005] In daily data development work, you need to use the following Figure 1 A Universe data development platform is shown for data set file development. The platform can create corresponding business data flows or workflows for enterprise users. Developers use the data development system of the platform to develop corresponding business data flows or workflows for enterprises and other users. During the data development process, developers need to synchronize and store the data files generated by the development into the database.

[0006] In development work, the preservation of development data sets is crucial and needs to be saved in real time to the platform database. However, the platform generally uses a fixed mode to synchronize and save data, such as using offline and batch synchronization tools based on Select queries such as DataX and Sqoop to synchronize the generated business data sets. However, there are the following problems: The platform uses a unified offline and batch synchronization tool to synchronize and save the development data set of each developer, which will cause multiple people to call the offline and batch synchronization tools online at the same time, occupying memory and processes, causing operating pressure and the security of backend data operation in various places. However, the workload of each person is different, so the platform uses a unified offline and batch synchronization tool for data synchronization, which cannot change with the amount of business data generated by development work. Summary of the invention

[0007] In order to solve the above problems, the present application proposes a method, system and electronic device for offline synchronization of incremental data in data development.

[0008] On the one hand, the present application proposes a method for offline synchronization of incremental data in data development, comprising the following steps: Write a synchronization program script and configure it in the data management service, and configure a data incremental synchronization tool in the data management service; Collecting business data sets generated by various developers during data set development, and calculating the data volume of the business data sets; Determine whether the data volume of the business data set exceeds a preset value: If it exceeds, the data incremental synchronization tool is called to perform incremental data offline synchronization on the business data set; Otherwise, the synchronization program script is called to perform offline synchronization of incremental data on the business data set.

[0009] As an optional implementation scheme of the present application, optionally, the data incremental synchronization tool adopts a real-time streaming synchronization method based on the database data change log, monitors the incremental data that has changed in the database, and synchronizes the incremental data to the database of the Universe data development platform according to a preset offline synchronization strategy.

[0010] As an optional implementation scheme of the present application, optionally, the data incremental synchronization tool adopts a real-time streaming synchronization tool based on the database data change log: Maxwell or Cana.

[0011] As an optional implementation scheme of the present application, optionally, collecting the business data sets generated by each developer in the data set development and calculating the data volume of the business data sets includes: According to the preset scheduling cycle, the business data sets generated by each developer in the data set development are scheduled from the database of the Universe data development platform and forwarded to the data management service; The data source docking module in the data management service receives the business data sets generated by developers during data set development and forwards them to the data set management module; The data set management module binds each business data set with the identity ID of the corresponding developer. After binding, the data volume of the development data contained in each business data set is calculated to obtain the data volume of each business data set, and the data volume of each business data set is bound to the identity ID of the corresponding developer.

[0012] As an optional implementation scheme of the present application, optionally, collecting the business data sets generated by each developer in the data set development and calculating the data volume of the business data sets also includes: In the data set management module, a business volume table is pre-configured to record the data volume of each developer; The business volume table includes: The header is used to record the developer's identity ID; The table item is used to record the data volume of the business data set generated by the developer during data set development.

[0013] As an optional implementation scheme of the present application, optionally, determining whether the data volume of the service data set exceeds a preset value includes: Traverse the business volume table, obtain the data volume of the business data set of each developer, and compare the data volume of the business data set of each developer with the preset value one by one, determine whether the data volume of the business data set of each developer exceeds the preset value, and output the identity ID of the developer who exceeds the preset value.

[0014] As an optional implementation scheme of the present application, optionally, when calling the data increment synchronization tool or the synchronization program script, the corresponding data increment synchronization tool or the synchronization program script is called through a task scheduling module according to the output result; According to the output identity ID of the developer that exceeds the preset value, the business data set bound to the identity ID is called, and the business data set is imported into the called data incremental synchronization tool or the synchronization program script, and the incremental data of the business data set is synchronized offline using the data incremental synchronization tool or the synchronization program script, and the data is synchronously saved to the database of the Universe data development platform.

[0015] On the other hand, the present application proposes a system for implementing the incremental data offline synchronization method in the data development, comprising: Configuration module, used to configure the written synchronization program script and data incremental synchronization tool in the data management service; The data management module is used to collect the business data sets generated by each developer in the data set development and calculate the data volume of the business data sets; Determine whether the data volume of the business data set exceeds a preset value: If it exceeds, the data incremental synchronization tool is called to perform incremental data offline synchronization on the business data set; Otherwise, the synchronization program script is called to perform offline synchronization of incremental data on the business data set.

[0016] In another aspect, the present application further provides an electronic device, comprising: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the incremental data offline synchronization method in data development when executing the executable instructions.

[0017] Technical effects of the present invention: This application determines whether to call the preset data incremental synchronization tool (Maxwell or Cana, based on the database data change log, to perform real-time streaming synchronization of data sets with data changes. For example, MySQL's binlog will record all insert, update, and delete operations in real time) by comparing the threshold value according to the amount of data in the business table of each developer: If the amount is large, the data incremental synchronization tool is called: Otherwise, the data is synchronized using a written script. Therefore, by flexibly calling the incremental synchronization tool according to the value of the business data set, it is possible to avoid frequent scheduling of a fixed incremental synchronization tool, thereby reducing the frequent occupation of the background running memory by the incremental synchronization tool, reducing the data running pressure, and improving the security of the background running system. At the same time, incremental synchronization and updating of data can be achieved to ensure data consistency and real-time performance.

[0018] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0020] Figure 1 Shown is an architecture diagram of the Universe data development platform of the present invention; Figure 2 It is shown as a schematic diagram of the implementation process of the present invention; Figure 3 Shown is a schematic diagram of tool / program script calling of the present invention; Figure 4 It is a schematic diagram showing the application of the electronic device of the present invention. DETAILED DESCRIPTION

[0021] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0022] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0023] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure. Example 1

[0024] like Figure 2 As shown, on the one hand, the present application proposes a method for offline synchronization of incremental data in data development, comprising the following steps: S1. Write a synchronization program script and configure it in the data management service, and configure a data incremental synchronization tool in the data management service; S2, collecting the business data sets generated by each developer in the data set development, and calculating the data volume of the business data sets; S3. Determine whether the data volume of the business data set exceeds a preset value: If it exceeds, the data incremental synchronization tool is called to perform incremental data offline synchronization on the business data set; Otherwise, the synchronization program script is called to perform offline synchronization of incremental data on the business data set.

[0025] The data synchronization of this solution will be determined according to the amount of business data, and a flexible data synchronization solution will be adopted. If the amount of business table data is relatively large and the proportion of daily data changes is relatively low, then the incremental synchronization of the real-time streaming synchronization tool will be used, otherwise the synchronization program script will be run for incremental synchronization. By flexibly calling the incremental synchronization tool according to the value of the business data set, it is possible to avoid frequent scheduling of a fixed incremental synchronization tool, thereby reducing the frequent occupation of the background running memory by the incremental synchronization tool, reducing the pressure of data operation, and improving the security of the background running system. At the same time, it can realize incremental synchronization and update of data to ensure data consistency and real-time performance.

[0026] Among them, there are two main types of offline data synchronization: full synchronization and incremental synchronization.

[0027] 1. Full synchronization is to synchronize all the data in the business database to the data warehouse every day. This is the simplest way to ensure data synchronization on both sides. For a table that uses full synchronization, it is usually necessary to perform a full synchronization on the first day.

[0028] 2. Incremental synchronization means synchronizing only the new and changed data in the business data to the data warehouse every day. For tables that use daily incremental synchronization, it is usually necessary to perform a full synchronization on the first day. If the business table data volume is relatively large and the proportion of daily data changes is relatively low, incremental synchronization should be used at this time, otherwise full synchronization can be used.

[0029] Data synchronization tools can be roughly divided into two categories: one is offline, batch synchronization tools based on Select queries, such as DataX and Sqoop; the other is real-time streaming synchronization tools based on database data change logs (such as MySQL's binlog, which records all insert, update, and delete operations in real time), such as Maxwell and Canal.

[0030] In this embodiment, considering that the data volume of the business data set generated by each developer in the data set development is different, in order to avoid causing greater operating pressure, it is not convenient to perform full synchronization according to the unified query-based offline synchronization tools such as DataX and Sqoop. Therefore, a real-time streaming synchronization tool based on database data change log (such as MySQL's binlog, which records all insert, update and delete operations in real time) represented by Maxwell and Canal is adopted to synchronize the incremental data in the data source (the business data set generated by each developer in the data set development) to the data target system (the platform's database).

[0031] As an optional implementation scheme of the present application, optionally, the data incremental synchronization tool adopts a real-time streaming synchronization method based on the database data change log, monitors the incremental data that has changed in the database, and synchronizes the incremental data to the database of the Universe data development platform according to a preset offline synchronization strategy.

[0032] As an optional implementation scheme of the present application, optionally, the data incremental synchronization tool adopts a real-time streaming synchronization tool based on the database data change log: Maxwell or Cana.

[0033] The specific working principle of the data incremental synchronization tool: (1) Incremental data capture and incremental recording (marking): (2) Timestamp / version number: Add a timestamp or version number field to each record in the source data. Whenever a record changes, update its timestamp or increase its version number. When synchronizing, only select records with timestamps later than the last synchronization time, or select records with version numbers higher than the last synchronization version.

[0034] (3) Log capture: Use the database’s change data capture function, such as MySQL’s binlog or Oracle’s redo log, to capture logs of data changes. Analyze the logs, extract incremental data, and synchronize them.

[0035] (4) Mark changes: Add a field to the data table, such as ischanged, with a default value of 0. When data changes, set ischanged to 1. When synchronizing, select the records with ischanged set to 1 and reset them to 0 after synchronization.

[0036] As an optional implementation scheme of the present application, optionally, collecting the business data sets generated by each developer in the data set development and calculating the data volume of the business data sets includes: According to the preset scheduling cycle, the business data sets generated by each developer in the data set development are scheduled from the database of the Universe data development platform and forwarded to the data management service; The data source docking module in the data management service receives the business data sets generated by developers during data set development and forwards them to the data set management module; The data set management module binds each business data set with the identity ID of the corresponding developer. After binding, the data volume of the development data contained in each business data set is calculated to obtain the data volume of each business data set, and the data volume of each business data set is bound to the identity ID of the corresponding developer.

[0037] During the data set development process, the system records the business data sets generated by each developer in real time and stores them in the platform's database in real time.

[0038] For details on the service functions of the data development platform, please refer to the attached Figure 1 The technical architecture shown includes data governance services, one-stop development services, and basic services. Developers can log in to the platform and use basic services to manage data sets and connect to data sources. On the one-stop development platform, development work can be carried out. Administrators can use basic services to schedule development tasks, create tasks, set task scheduling cycles and priorities, and distribute the set development tasks to individual developers.

[0039] Administrators can set the scheduling cycle for the business data generated by each developer in the task scheduling module. After setting, the task scheduling module is used to uniformly and regularly retrieve each business data set according to the set scheduling cycle. The corresponding business data set is retrieved from the platform's database according to the identity ID of each person, and forwarded to the data management module through the API interface. The data source docking module of the data management module receives the data and forwards it to the data set management module, where the collected business data set can be bound to an ID. The data set management module also performs calculation tasks, calculates the data volume of the development data contained in each business data set, obtains the data volume of the corresponding business data set, and binds the data volume to the identity ID of each developer, so that the size value of the business volume of each developer is obtained under this sampling.

[0040] As an optional implementation scheme of the present application, optionally, collecting the business data sets generated by each developer in the data set development and calculating the data volume of the business data sets also includes: In the data set management module, a business volume table is pre-configured to record the data volume of each developer; The business volume table includes: The header is used to record the developer's identity ID; The table item is used to record the data volume of the business data set generated by the developer during data set development.

[0041] As an optional implementation scheme of the present application, optionally, determining whether the data volume of the service data set exceeds a preset value includes: Traverse the business volume table, obtain the data volume of the business data set of each developer, and compare the data volume of the business data set of each developer with the preset value one by one, determine whether the data volume of the business data set of each developer exceeds the preset value, and output the identity ID of the developer who exceeds the preset value.

[0042] In order to facilitate the statistics of each person's business volume and to facilitate rapid traversal, comparison and calculation, this solution pre-sets a business volume table in the data-level management module to count and record the data volume of each developer: the header is used to register the identity ID of each developer, and the subsequent table items are used to count the data volume of each developer.

[0043] When calculating the quantity value later, the traversal algorithm is used to traverse each data quantity value in the business quantity table, and the data set management module calls the preset quantity value to compare with each data quantity. According to the comparison result, the corresponding data incremental synchronization tool or synchronization program script is called to perform incremental synchronization processing on the corresponding data set.

[0044] For example, if the amount of data exceeds the preset value, the incremental data synchronization tool will be called to perform incremental synchronization processing for business data sets that exceed the preset value. For business data sets with smaller data volumes, the synchronization program script can be directly called for processing. There is no need to use tools with larger running memory to continuously occupy the line for processing. The program script with smaller memory can be used for data synchronization processing.

[0045] As an optional implementation scheme of the present application, optionally, when calling the data increment synchronization tool or the synchronization program script, the corresponding data increment synchronization tool or the synchronization program script is called through a task scheduling module according to the output result; According to the output identity ID of the developer that exceeds the preset value, the business data set bound to the identity ID is called, and the business data set is imported into the called data incremental synchronization tool or the synchronization program script, and the incremental data of the business data set is synchronized offline using the data incremental synchronization tool or the synchronization program script, and the data is synchronously saved to the database of the Universe data development platform.

[0046] Technical solution for data incremental synchronization and update, specific implementation method: 1. Data source preparation: Determine the data source that needs to be synchronized, such as collecting the business data sets generated by various developers during data set development through the data processing module / database and data warehouse API interface.

[0047] 2. Incremental data capture and incremental recording (marking): Timestamp / Version Number: Add a timestamp or version number field to each record in the source data. Whenever a record changes, update its timestamp or increase its version number. When synchronizing, only select records with timestamps later than the last synchronization time, or select records with version numbers higher than the last synchronization version.

[0048] Log capture: Use the database's change data capture function, such as MySQL's binlog or Oracle's redolog, to capture logs of data changes. Analyze the logs, extract incremental data, and synchronize.

[0049] Mark changes: Add a field to the data table, such as ischanged, with a default value of 0. When data changes, set ischanged to 1. When synchronizing, select the records with ischanged set to 1 and reset them to 0 after synchronization.

[0050] 3. Calculate the data volume and compare the values ​​of business data sets, find business data sets with values ​​exceeding the standard, and import them into the tool for incremental synchronization.

[0051] 4. Data incremental synchronization tools: Real-time streaming synchronization tools based on database data change logs (such as MySQL binlog, which records all insert, update, and delete operations in real time), such as Maxwell and Canal. Or use ETL tools to help extract, transform, and load data, monitor database changes, and perform incremental synchronization. Incremental data in the data source is synchronized to the data target system.

[0052] 4.1 Synchronization program scripts, including synchronization programs or synchronization scripts for incremental synchronization: such as Python scripts, Java programs, etc., can be used to implement incremental data monitoring, incremental data detection and synchronization by writing scripts or programs. The specific programs or scripts can be configured by developers or administrators. The writing and configuration of mini-programs / scripts can refer to the processing flow of the data incremental synchronization tool used to implement a processing program with similar functions.

[0053] 4.2 Target data storage: Determine the database of the target data storage Universe data development platform.

[0054] 5. Synchronization strategy: Periodic synchronization: Set a fixed synchronization interval, such as 2 a.m. every day. During each synchronization, record the synchronization time, and during the next synchronization, only synchronize the data from the last recorded time to the present.

[0055] Real-time synchronization: Use real-time data synchronization tools, such as Apache Kafka and Apache Flink, to achieve real-time data synchronization.

[0056] Data synchronization will be determined based on the amount of business data, and a flexible data synchronization solution will be adopted. If the amount of business table data is relatively large and the proportion of daily data changes is relatively low, then the incremental synchronization of the real-time streaming synchronization tool will be used. Otherwise, the synchronization program script will be run for incremental synchronization. By flexibly calling the incremental synchronization tool based on the value of the business data set, it is possible to avoid frequent scheduling of a fixed incremental synchronization tool, thereby reducing the frequent occupation of the background running memory by the incremental synchronization tool, reducing the pressure of data operation, and improving the security of the background running system. At the same time, it can realize incremental synchronization and update of data to ensure data consistency and real-time performance.

[0057] 6. System monitoring and maintenance: Monitoring: Monitor the data synchronization process to ensure data consistency and integrity.

[0058] Log: Record the logs during the synchronization process to facilitate troubleshooting and performance optimization.

[0059] Permission control: Strictly control access rights to source and target data to ensure data security.

[0060] 7.Performance optimization: Asynchronous processing: Use asynchronous message queues (such as Kafka) to temporarily store incremental data and reduce database pressure during the synchronization process.

[0061] Obviously, those skilled in the art should understand that the implementation of all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory. Example 2

[0062] Based on the implementation principle of Example 1, on the other hand, the present application proposes a system for implementing the incremental data offline synchronization method in the data development, including: Configuration module, used to configure the written synchronization program script and data incremental synchronization tool in the data management service; The data management module is used to collect the business data sets generated by each developer in the data set development and calculate the data volume of the business data sets; Determine whether the data volume of the business data set exceeds a preset value: If it exceeds, the data incremental synchronization tool is called to perform incremental data offline synchronization on the business data set; Otherwise, the synchronization program script is called to perform offline synchronization of incremental data on the business data set.

[0063] The above configuration module mainly uses the data configuration function of the platform to provide configuration services for administrators. Administrators can use the data set management center of the Universe platform to perform data configuration.

[0064] Data source connection module and data set management module, please refer to the attached Figure 1The technical principles and interactions of the data management service, etc. in the embodiment can be understood by referring to the description in Example 1, and will not be repeated here.

[0065] The modules or steps of the present invention described above can be implemented by a general-purpose computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software. Example 3

[0066] Furthermore, in another aspect, the present application also proposes an electronic device, comprising: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the incremental data offline synchronization method in data development when executing the executable instructions.

[0067] The electronic device according to the embodiment of the present disclosure includes a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement any of the above-mentioned incremental data offline synchronization methods in data development when executing the executable instructions.

[0068] Here, it should be noted that the number of processors can be one or more. At the same time, the electronic device of the embodiment of the present disclosure may also include an input system and an output system. Among them, the processor, memory, input system and output system may be connected through a bus or in other ways, which are not specifically limited here.

[0069] The memory, as a computer-readable storage medium, can be used to store software programs, computer executable programs and various modules, such as the program or module corresponding to the incremental data offline synchronization method in data development in an embodiment of the present disclosure. The processor executes various functional applications and data processing of the electronic device by running the software programs or modules stored in the memory.

[0070] The input system can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output system can include display devices such as display screens.

[0071] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for offline synchronization of incremental data in data development, characterized in that: The steps include: Write a synchronization program script and configure it in the data management service, and configure a data incremental synchronization tool in the data management service; Collecting business data sets generated by various developers during data set development, and calculating the data volume of the business data sets; Determine whether the data volume of the business data set exceeds a preset value: If it exceeds, the data incremental synchronization tool is called to perform incremental data offline synchronization on the business data set; Otherwise, the synchronization program script is called to perform offline synchronization of incremental data on the business data set.

2. According to the method for offline synchronization of incremental data in data development in claim 1, it is characterized in that: The data incremental synchronization tool adopts a real-time streaming synchronization method based on the database data change log, monitors the incremental data that has changed in the database, and synchronizes the incremental data to the database of the Universe data development platform according to a preset offline synchronization strategy.

3. The incremental data offline synchronization method in data development according to claim 2 is characterized in that: The data incremental synchronization tool adopts a real-time streaming synchronization tool based on the database data change log: Maxwell or Cana.

4. The incremental data offline synchronization method in data development according to claim 1 is characterized in that: Collect the business data sets generated by each developer during data set development, and calculate the data volume of the business data sets, including: According to the preset scheduling cycle, the business data sets generated by each developer in the data set development are scheduled from the database of the Universe data development platform and forwarded to the data management service; The data source docking module in the data management service receives the business data sets generated by developers during data set development and forwards them to the data set management module; The data set management module binds each business data set with the identity ID of the corresponding developer. After binding, the data volume of the development data contained in each business data set is calculated to obtain the data volume of each business data set, and the data volume of each business data set is bound to the identity ID of the corresponding developer.

5. The incremental data offline synchronization method in data development according to claim 1 is characterized in that: Collecting the business data sets generated by various developers in the data set development and calculating the data volume of the business data sets also includes: In the data set management module, a business volume table is pre-configured to record the data volume of each developer; The business volume table includes: The header is used to record the developer's identity ID; The table item is used to record the data volume of the business data set generated by the developer during data set development.

6. The incremental data offline synchronization method in data development according to claim 5 is characterized in that: Determining whether the data volume of the business data set exceeds a preset value includes: Traverse the business volume table, obtain the data volume of the business data set of each developer, and compare the data volume of the business data set of each developer with the preset value one by one, determine whether the data volume of the business data set of each developer exceeds the preset value, and output the identity ID of the developer who exceeds the preset value.

7. The incremental data offline synchronization method in data development according to claim 6 is characterized in that: When calling the data increment synchronization tool or the synchronization program script, the corresponding data increment synchronization tool or the synchronization program script is called through the task scheduling module according to the output result; According to the output identity ID of the developer that exceeds the preset value, the business data set bound to the identity ID is called, and the business data set is imported into the called data incremental synchronization tool or the synchronization program script, and the incremental data of the business data set is synchronized offline using the data incremental synchronization tool or the synchronization program script, and the data is synchronously saved to the database of the Universe data development platform.

8. A system for implementing the incremental data offline synchronization method in data development as described in any one of claims 1 to 7, characterized in that: include: Configuration module, used to configure the written synchronization program script and data incremental synchronization tool in the data management service; The data management module is used to collect the business data sets generated by each developer in the data set development and calculate the data volume of the business data sets; Determine whether the data volume of the business data set exceeds a preset value: If it exceeds, the data incremental synchronization tool is called to perform incremental data offline synchronization on the business data set; Otherwise, the synchronization program script is called to perform offline synchronization of incremental data on the business data set.

9. An electronic device, characterized in that include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement an incremental data offline synchronization method in data development according to any one of claims 1-7 when executing the executable instructions.

Citation Information

Cited By

  • AI-based enterprise business data processing system and method

    CN119917495A