A method for guaranteeing consistency of mass real-time storage low-delay full data synchronization

Abstract

The present application relates to the technical field of data caching, and more particularly to a method for guaranteeing consistency of massive real-time storage and low-delay full-amount data synchronization, comprising: writing incremental data into a message queue by using a data synchronization tool to generate a checkpoint; setting a target real-time storage library to an offline state; performing data conversion processing on full-amount latest data and importing the converted data into the target real-time storage library; setting the target real-time storage library to an online state; starting a CDC incremental data processing flow program, consuming incremental data in the message queue, and performing playback processing on the converted data stored in the target real-time storage library. The present application can reduce the delay impact on the data of the target real-time storage library that has been put online by introducing an effective incremental synchronization mechanism, so as to capture the data changes of the source business system database and quickly apply these changes to the target real-time storage library, which is helpful for timely decision-making and responding to market changes.

Description

Technical Field

[0001] This invention relates to the field of data caching technology, and in particular to a method for ensuring the consistency of full data synchronization in massive real-time storage with low latency. Background Technology

[0002] In applications with high real-time requirements, real-time computation using complete historical data is necessary. Due to the massive scale of this data, it is typically stored in a data warehouse, which generally uses a distributed file system (such as HDFS). However, the characteristics of HDFS's file storage structure make it unable to meet the performance requirements of certain online real-time services when working with upper-layer computing engines. Therefore, to solve this problem, this large-scale data is usually migrated to a real-time repository. Although this real-time repository uses a heterogeneous storage method from the data warehouse, it provides better support in terms of real-time performance.

[0003] Furthermore, data stored in offline data warehouses typically lags behind the latest business data. This is because most offline warehouses import business system data from a specific point in time through full extraction, with relatively long extraction intervals, which cannot meet the needs of real-time scenarios. However, to ensure the consistency of real-time data, incremental data needs to be updated quickly to the real-time repository in a low-latency manner. This incremental update method ensures that the data stored in real-time is synchronized with the data in the source business system database and meets the requirements of real-time analytics applications, thus providing better consistency.

[0004] In the process of migrating data from the source business system database to real-time incremental storage, the SQOOP tool and JDBC are usually used to connect to the source business system database and import the full data into the real-time storage system. However, the existing technology has some limitations, such as: (1) For some large data tables, the full data import operation usually takes a long time, and in some cases, the data of these large tables cannot be completely extracted. This will result in a significant lag in the availability of data in the real-time storage system, which cannot meet the needs of real-time data. At the same time, this delay defect is not suitable for real-time business scenarios that require immediate access and analysis of data; (2) Large-scale data extraction operations may cause the performance of the source business system database to degrade, which will have a negative impact on online business. At the same time, there is a potential risk to maintaining the stability and availability of the source business system; (3) There may be inconsistencies in the data during the full data import process. If the data in the source business system database changes during the full data import process, it may lead to data inconsistencies. Additional maintenance and processing work is required to ensure the accuracy and consistency of the data.

[0005] In summary, given the problems of existing technologies, such as the source business system database lacking high throughput, the potential impact of large-scale data extraction on online business, and the low efficiency and high latency of large-scale data extraction, it is necessary to design a method to ensure the consistency of full data synchronization in massive real-time storage with low latency. This is to meet the urgent need for data real-time performance and consistency, while reducing the impact on the source business system database and the target real-time storage system. Summary of the Invention

[0006] Therefore, it is necessary to provide a method to ensure the consistency of full data synchronization in massive real-time storage with low latency, in response to the above-mentioned technical problems.

[0007] According to one aspect of the present invention, a method for ensuring the consistency of low-latency full data synchronization in massive real-time storage is provided, comprising: acquiring incremental data from a source business system database; writing the incremental data into a message queue using a data synchronization tool to obtain CDC log data under a target topic in the message queue; and generating checkpoints for the corresponding CDC log data based on the CDC log data; setting the target real-time repository to an offline state according to the actual need to bring the source business system database online to the target real-time repository; performing data transformation processing on the latest full data using Spark SQL or Flink SQL tools to obtain transformed data, and importing the transformed data into the target real-time repository; setting the target real-time repository to an online state after the transformed data has been imported; starting a CDC incremental data processing stream program; consuming incremental data in the message queue according to the order of checkpoints under the target topic in the message queue; and performing replay processing on the transformed data stored in the target real-time repository.

[0008] In some optional implementations of certain embodiments, the step of obtaining incremental data from the source business system database, writing the incremental data to a message queue using a data synchronization tool to obtain CDC log data under the target topic in the message queue, and generating checkpoints for the corresponding CDC log data based on the CDC log data specifically includes: obtaining multiple incremental data from the source business system database, writing each incremental data to a message queue using CDC middleware to obtain multiple CDC log data under the target topic in the message queue, and generating checkpoints for the corresponding CDC log data based on each CDC log data.

[0009] In some optional implementations of certain embodiments, the state of the target real-time repository includes an online state and an offline state. If the target real-time repository is set to an online state, the CDC incremental data processing stream program performs data transformation processing on the incremental data according to the JSON format of the data synchronization tool, and imports the data-transformed incremental data into the target real-time repository. If the target real-time repository is set to an offline state, the CDC incremental data processing stream program will not import the received incremental data into the target real-time repository.

[0010] In some optional implementations of certain embodiments, the full latest data includes the full historical data of the offline data warehouse and the daily incremental data of the source business system database.

[0011] In some optional implementations of certain embodiments, the step of using Spark SQL or Flink SQL tools to perform data transformation processing on the full set of latest data to obtain transformed data, and then importing the transformed data into the target real-time repository, specifically includes: using Spark SQL or Flink SQL tools to query the full set of historical data in the offline data warehouse and the daily incremental data in the source business system database to extract the full set of latest data; based on the mapping relationship formed between the full set of latest data and the data types of the target real-time repository, performing data transformation processing on the full set of latest data to obtain transformed data; and then using the Spark distributed engine or Flink distributed engine to import the transformed data into the target real-time repository.

[0012] In some optional implementations of certain embodiments, before the step of starting the CDC incremental data processing stream program, the method further includes: obtaining the checkpoint of the first CDC log data of the day under the target topic in the message queue, and starting the CDC incremental data processing stream program based on the checkpoint recorded under the target topic in the message queue.

[0013] In some optional implementations of certain embodiments, the step of starting the CDC incremental data processing stream program, consuming incremental data in the message queue according to the order of checkpoints under the target topic in the message queue, and replaying the transformation processing data stored in the target real-time repository, specifically includes: starting the CDC incremental data processing stream program, consuming incremental data in the message queue according to the order of checkpoints under the target topic in the message queue, and replaying the transformation processing data stored in the target real-time repository according to the log type of the CDC log data, until the latest checkpoint under the target topic in the message queue is consumed.

[0014] In some optional implementations of certain embodiments, the log types of the CDC log data include: insertion logs, update logs, and deletion logs.

[0015] According to a second aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0016] According to a third aspect of the present invention, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of the above-described method.

[0017] The advantages and beneficial effects of this invention are as follows: This invention provides a method for ensuring the consistency of full data synchronization in massive real-time storage with low latency. This method can overcome the problem that large-scale data tables in the source business system database cannot be efficiently uploaded to the target real-time storage repository. It comprehensively considers the performance and resource optimization in the data extraction, transformation, and loading processes. It not only ensures data integrity during migration but also reduces the time cost and resource consumption of migration operations. By introducing an effective incremental synchronization mechanism, it can reduce the impact of data latency on the already uploaded target real-time storage repository, capture data changes in the source business system database, and quickly apply these changes to the target real-time storage repository. This allows for the acquisition of the latest incremental data in a short time, which helps in timely decision-making and response to market changes. In addition, it also comprehensively considers concurrent access and high availability to ensure that the target real-time storage repository can meet high-throughput requests without being affected by latency. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for ensuring the synchronization and consistency of massive real-time storage of low-latency full data in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of an electronic device in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Example 1

[0022] Appendix Figure 1 The flowchart illustrates a method for ensuring the consistency of full data synchronization in massive real-time storage with low latency, as provided in this application.

[0023] Reference Appendix Figure 1 A method for ensuring the consistency of full data synchronization in real-time storage with low latency is proposed, which includes the following steps.

[0024] S1. Obtain incremental data from the source business system database, use a data synchronization tool to write the incremental data to a message queue, obtain CDC log data under the target topic in the message queue, and generate checkpoints for the corresponding CDC log data based on the CDC log data.

[0025] In some embodiments, the CDC (Change Data Capture) incremental data processing stream program of the present invention can utilize common stream processing frameworks in the big data field, such as Flink, Spark Streaming, Structured Streaming, etc., to implement the operation of driving metadata to write to a target real-time repository.

[0026] In some embodiments, when the CDC incremental data processing stream is first launched, it consumes data from the message queue in a "latest" manner. This helps avoid consuming a large amount of message queue data at once, thereby reducing data latency at the initial deployment stage. Once the first batch of data is successfully written to the target real-time repository and a valid checkpoint is generated, subsequent launches of the CDC incremental data processing stream can continue to consume data from the message queue based on the successfully committed checkpoint (i.e., offset), thus ensuring that data is not lost. This strategy helps maintain data continuity and integrity, ensuring that real-time data remains up-to-date in storage. It should be noted that the message queues involved in this invention include, but are not limited to, Kafka, Pulsar, Redis, etc., which can be selected according to actual usage. In this application, Kafka is used as an example for illustration.

[0027] In some embodiments, to ensure the integrity of incremental data, it is necessary to store the incremental data in Kafka first to avoid the problem of missing incremental data in the daily operations.

[0028] In some embodiments, the step of obtaining incremental data from the source business system database, using a data synchronization tool to write the incremental data into a message queue, obtaining CDC log data under the target topic in the message queue, and generating checkpoints for the corresponding CDC log data based on the CDC log data, specifically includes: obtaining multiple incremental data from the source business system database, using CDC middleware to write each incremental data into a message queue, obtaining multiple CDC log data under the target topic in the message queue, and generating checkpoints for the corresponding CDC log data based on each CDC log data.

[0029] In some embodiments, the data synchronization tools involved in this invention include, but are not limited to, Canal middleware, OGG middleware, etc., which can be selected according to the actual use case, and are not limited here.

[0030] S2. Based on the actual need to bring the source business system database online to the target real-time repository, set the target real-time repository to an offline state.

[0031] In some embodiments, when a request is received to bring a source business system database online to a target real-time repository, for example, when a business analyst needs to make decisions based on near real-time data, the complete and latest full data needs to be initialized to the target real-time repository. It should be noted that when performing the data go-live operation, care must be taken not to stop the CDC incremental data processing stream program that is already online and running. Stopping the current CDC incremental data processing stream program will cause all target tables associated with it to stop receiving updates, which will seriously affect real-time analysis applications that depend on these target tables and cause data delay problems.

[0032] In some embodiments, the state of the target real-time repository includes an online state and an offline state. If the target real-time repository is set to an online state, the CDC incremental data processing stream program performs data transformation processing on the incremental data according to the JSON format of the data synchronization tool, and imports the data-transformed incremental data into the target real-time repository. If the target real-time repository is set to an offline state, the CDC incremental data processing stream program will not import the received incremental data into the target real-time repository.

[0033] In some embodiments, in the target real-time repository, the metadata of each table includes an online status and an offline status. When the table metadata is in an offline status, the CDC incremental data processing stream program will ignore the data import operation for that table. This can avoid the problem of data inconsistency when the CDC incremental data processing stream program and the full import program write data at the same time.

[0034] S3. Use Spark SQL or Flink SQL tools to transform the latest full dataset, obtain the transformed data, and import the transformed data into the target real-time repository.

[0035] In some embodiments, the CDC incremental data processing stream program performs data transformation processing according to the JSON format of different CDC middleware and writes the transformed data to the target real-time repository.

[0036] In some embodiments, the full latest data includes the full historical data of the offline data warehouse and the daily incremental data of the source business system database.

[0037] In some embodiments, Spark SQL or Flink SQL tools are used to transform the full set of latest data to obtain transformed data, and the transformed data is then imported into the target real-time repository. Specifically, this includes: using Spark SQL or Flink SQL tools to query the full set of historical data in the offline data warehouse and the daily incremental data in the source business system database to extract the full set of latest data; based on the mapping relationship between the full set of latest data and the data types of the target real-time repository, performing data transformation on the full set of latest data to obtain transformed data; and using the Spark distributed engine or Flink distributed engine to import the transformed data into the target real-time repository.

[0038] In some embodiments, to avoid interfering with the source business system database and to avoid using tools like SQOOP to extract data from the source business system database, especially for large tables with more than 10 billion rows, tools such as Spark SQL or Flink SQL can be used to query a high-throughput data warehouse (i.e., an offline data warehouse and the source business system database) to extract the full and latest data. Then, through a distributed engine such as Spark or Flink, this large-scale data is loaded into the target real-time repository. This method not only avoids the burden on the source business system database but also meets the processing needs of large-scale data, while effectively reducing interference with online business.

[0039] S4. After the conversion and processing data import is complete, set the target real-time repository to online status.

[0040] In some embodiments, when bringing a target real-time repository online, it is necessary to ensure that the target real-time repository is in an offline state. This requires that each table stored in the target real-time repository has updatable state information, which can be manually updated by the user when bringing near-real-time tables online. For example, when importing all the latest data, the target real-time repository is set to an offline state. Once the import of all the latest data is complete, the target real-time repository is switched to an online state. The state switching mechanism of the target real-time repository ensures that the running CDC incremental data processing stream program will not write the received CDC messages (i.e., incremental data stored in Kafka) to the target real-time repository, thereby preventing interference and overwriting with the subsequently started full import program and avoiding data inconsistency issues.

[0041] S5. Obtain the checkpoint of the first CDC log data of the day under the target topic in the message queue, and start the CDC incremental data processing stream program based on the checkpoint recorded under the target topic in the message queue.

[0042] In some embodiments, when the target real-time repository is brought online, it is first necessary to obtain the checkpoint of the first CDC log data of the day in the message queue. The purpose of this step is that once the full initialization and incremental extraction of real-time storage data are completed, the checkpoint recorded in the message queue can be used to start the CDC incremental data processing stream program to ensure that the incremental data of the day can be correctly replayed into the target real-time repository in order to maintain the integrity and timeliness of the data.

[0043] S6. Start the CDC incremental data processing stream program. According to the order of checkpoints under the target topic in the message queue, the CDC incremental data processing stream program consumes the incremental data in the message queue and performs replay processing on the transformation processing data stored in the target real-time repository.

[0044] In some embodiments, starting the CDC incremental data processing stream program, consuming incremental data in the message queue according to the order of checkpoints under the target topic in the message queue, and performing replay processing on the transformation processing data stored in the target real-time repository, specifically includes: starting the CDC incremental data processing stream program, consuming incremental data in the message queue according to the order of checkpoints under the target topic in the message queue, and performing replay processing on the transformation processing data stored in the target real-time repository according to the log type of the CDC log data, until the latest checkpoint under the target topic in the message queue is consumed.

[0045] In some embodiments, the log types of the CDC log data include: insertion logs, update logs, and deletion logs.

[0046] In some embodiments, when the full import program completes all data writing, the currently running CDC incremental data processing stream program needs to be stopped, and then the target real-time repository needs to be brought online. The CDC incremental data processing stream program is then started with the previously saved checkpoint. After starting the CDC incremental data processing stream program, the incremental data of the target real-time repository for the day is replayed to ensure that the data consistency in the target real-time repository is maintained and guaranteed. It should be noted that this process can ensure that the data in the target real-time repository is synchronized with the data in the source business system database and meets the requirements of real-time analysis applications.

[0047] In some embodiments, the CDC incremental data processing stream program performs log insertion, log update, or log deletion operations (i.e., INSERT / UPDATE / DELETE operations) in the target real-time repository according to the order of checkpoints, ensuring data consistency after execution.

[0048] In some embodiments, in a Hadoop-based data analysis scenario, taking the massive historical data accumulated in the source business system database as an example, the method involved in this application will be further explained. Here, the massive historical data, such as hundreds of billions of data, cannot be fully extracted and transmitted through the standard data interface. In order to achieve near real-time data analysis and ensure data integrity, the source business system database storing such massive historical data is initialized into the target real-time repository. The offline data warehouse usually extracts data incrementally through indexes. In order to import these data completely into the target real-time repository, the entire data online operation can be achieved through the collaboration of the offline data warehouse, the source business system database, checkpoints and Kafka. The specific operations include: (1) using SparkSQL tools or Flink The SQL tool processes the full historical data of the offline data warehouse, converts the data into the schema of the target real-time repository, and imports the processed data into the target real-time repository; (2) records the checkpoints of the incremental data to be launched under the target topic in the message queue, so that the CDC incremental data processing stream program can use it later; (3) uses the incremental data extraction method to completely extract the latest incremental data of the day into the target real-time repository; (4) sets the startup status of the CDC incremental data processing stream program, starts the CDC incremental data processing stream program with the checkpoint of the first CDC log data of the day, and starts consuming the incremental data in Kafka according to the order of the checkpoints, and performs replay processing on the transformation processing data stored in the target real-time repository according to the log type of the CDC log data (e.g., INSERT / UPDATE / DELETE operation) until the latest checkpoint under the target topic in Kafka is consumed.

[0049] Example 2

[0050] Based on Embodiment 1 described above, this embodiment also provides an electronic device, please refer to the appendix. Figure 2 , Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0051] like Figure 2As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0052] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 2 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively. Figure 2 Each box shown can represent a device or multiple devices as needed.

[0053] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0054] Example 3

[0055] Based on Embodiment 1 above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above method.

[0056] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), or any suitable combination thereof.

[0057] In this embodiment, the client and server can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0058] The aforementioned computer-readable medium may be included in the aforementioned device or may exist independently without being assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire training data and transform the training data to obtain initial data; determine an initial rule base based on the initial data and optimize the parameters of the initial rule base to obtain a target rule base; calculate activation weights for the rules in the target rule base according to a preset activation weight calculation formula; and determine abnormal information based on test data and the activation weights.

[0059] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0060] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0061] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a data acquisition unit, a rule determination unit, a weight calculation unit, and an anomaly determination unit. The names of these units do not necessarily limit the specific unit; for example, a data acquisition unit may also be described as a "unit for acquiring training data."

[0062] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0063] Obviously, those skilled in the art will understand that the various steps of the present invention described above can be performed in a manner different from that described above, and the simulation methods and experimental equipment include, but are not limited to, the above description. The steps of the present invention described above can be performed in a different order in certain circumstances, and the steps shown or described above can be performed separately. Therefore, the present invention is not limited to any particular combination of hardware and software.

[0064] The above description, in conjunction with specific embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for ensuring the consistency of low-latency full data synchronization in massive real-time storage, characterized in that, include: The incremental data is obtained from the source business system database. The incremental data is written to the message queue using a data synchronization tool. The CDC log data under the target topic in the message queue is obtained. Based on the CDC log data, the corresponding checkpoints of the CDC log data are generated. Based on the actual need to bring the source business system database online to the target real-time repository, the target real-time repository is set to offline status; The target real-time repository has two states: online and offline. If the target real-time repository is set to online, the CDC incremental data processing stream program performs data transformation on the incremental data according to the JSON format of the data synchronization tool, and imports the transformed incremental data into the target real-time repository. If the target real-time repository is set to offline, the CDC incremental data processing stream will not import the received incremental data into the target real-time repository. The Spark SQL or Flink SQL tools are used to transform the full and latest data to obtain transformed data, and then the transformed data is imported into the target real-time repository. Once the data conversion and processing are complete, set the target real-time repository to online status. The CDC incremental data processing stream program is started. Based on the order of checkpoints under the target topic in the message queue, the CDC incremental data processing stream program consumes incremental data in the message queue and performs replay processing on the transformation processing data stored in the target real-time repository.

2. The method for ensuring the synchronization and consistency of massive real-time storage of low-latency full data according to claim 1, characterized in that, The process of acquiring incremental data from the source business system database involves using a data synchronization tool to write the incremental data into a message queue, obtaining CDC log data under the target topic in the message queue, and generating checkpoints for the corresponding CDC log data based on the CDC log data. Specifically, this includes: acquiring multiple incremental data from the source business system database, using CDC middleware to write each incremental data into a message queue, obtaining multiple CDC log data under the target topic in the message queue, and generating checkpoints for the corresponding CDC log data based on each CDC log data.

3. The method for ensuring the synchronization and consistency of massive real-time storage of low-latency full data according to claim 1, characterized in that, The full and latest data includes the full historical data of the offline data warehouse and the daily incremental data of the source business system database.

4. The method for ensuring the synchronization and consistency of massive real-time storage of low-latency full data according to claim 3, characterized in that, The process of using Spark SQL or Flink SQL to transform the full set of latest data to obtain transformed data, and then importing the transformed data into the target real-time repository, specifically includes: using Spark SQL or Flink SQL to query the full historical data of the offline data warehouse and the daily incremental data of the source business system database to extract the full set of latest data; based on the mapping relationship between the full set of latest data and the data types of the target real-time repository, performing data transformation on the full set of latest data to obtain transformed data; and then using the Spark distributed engine or Flink distributed engine to import the transformed data into the target real-time repository.

5. The method for ensuring the synchronization and consistency of massive real-time storage of low-latency full data according to claim 1, characterized in that, Before the step of starting the CDC incremental data processing stream program, the method further includes: obtaining the checkpoint of the first CDC log data of the day under the target topic in the message queue, and starting the CDC incremental data processing stream program based on the checkpoint recorded under the target topic in the message queue.

6. The method for ensuring the synchronization and consistency of massive real-time storage of low-latency full data according to claim 1, characterized in that, The process of starting the CDC incremental data processing stream program involves consuming incremental data from the message queue according to the order of checkpoints under the target topic in the message queue, and replaying the transformation processing data stored in the target real-time repository. Specifically, this includes: starting the CDC incremental data processing stream program, consuming incremental data from the message queue according to the order of checkpoints under the target topic in the message queue, and replaying the transformation processing data stored in the target real-time repository according to the log type of the CDC log data, until the latest checkpoint under the target topic in the message queue is consumed.

7. The method for ensuring the synchronization and consistency of massive real-time storage of low-latency full data according to claim 6, characterized in that, The log types of the CDC log data include: insertion logs, update logs, and deletion logs.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images