Method and system for realizing real-time dynamic acquisition of data from multiple data centers to Dores based on FlinkCDC
Real-time data acquisition is realized in a multi-data center environment through FlinkCDC component, solving the problems of real-time acquisition, dynamic recovery, table structure changes, data encryption and heterogeneous data type conversion, real-time calculation and secure encryption of data, and dynamic management of Doris Synchronize tasks.
Patent Information
- Application Number
- CN202410996471.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-22
AI Technical Summary
Existing real-time data acquisition methods and systems cannot achieve real-time acquisition, dynamic recovery of interrupts, discovery of table structure changes, data encryption security, heterogeneous data type conversion and other needs, especially in multi-data center environments.
Using FlinkCDC components, the source system's binlog incremental log information is collected, and data synchronization information is configured, data flow is merged, and the DorisSink function is implemented based on Flink, processing data flow, real-time dynamic collection of data writing to Doris , supports data encryption, custom field addition and DDL change real-time playback.
Real-time data acquisition, real-time calculation, and consistent semantics are realized, Doris data synchronization components are optimized, and the dynamic addition and reduction of synchronization tasks are supported, ensuring data security and efficient synchronization.
Smart Images

Figure CN119003586A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data collection, and in particular to a method and system for implementing real-time dynamic collection of multi-data center data to Doris based on FlinkCDC. Background Art
[0002] In the current real-time data collection method and system, real-time data is written into Doris using the StreamLoad method. When a new table is added on the data source side, the corresponding table needs to be manually created in Doris; if the table structure of the table being synchronized changes on the data source side, it will cause synchronization errors and interruptions, and then the table structures on both ends need to be manually aligned before the synchronization can be restarted; in the case of cross-border data transmission, the data needs to meet the security requirements of the country where it is located, and the current real-time data collection method and system cannot encrypt the data while collecting it; Doris needs to label the data collected from different data centers, and the current real-time data collection method and system cannot meet the needs of adding additional labeling fields; when the data source and Doris are heterogeneous, the current real-time data collection method and system cannot automatically convert data types; when the synchronization table is divided into libraries and tables in the source system, it is necessary to configure the synchronization tasks of each library and table separately. In summary, the existing real-time data collection method and system cannot meet the needs of real-time collection, dynamic recovery of interruptions, discovery of table structure changes, data encryption security, and conversion of heterogeneous data types. Summary of the invention
[0003] According to an embodiment of the present invention, a method for implementing real-time dynamic collection of multi-data center data to Doris based on FlinkCDC is provided, comprising the following steps: Collect the incremental log information of the binlog of the source system and encapsulate the incremental log information into a DataStream data stream; Configure data synchronization information, use Flink to read data synchronization information and generate newAddStream data stream; Merge the DataStream data stream and the newAddStream data stream to generate the DataSyncStream data stream; Implement the DorisSink function based on Flink, process the DataSyncStream data stream according to the data synchronization information, and generate the updated DataSyncStream data stream; Write the data in the updated DataSyncStream data stream to Doris.
[0004] Further, configuring data synchronization information includes the following steps: Configure the source system connection information, table information, fields to be encrypted, and custom fields to be added required for synchronization; Determine whether the Doris system has a target table corresponding to the source system table. If not, obtain the table creation statement of the source system table and convert it into a Doris table creation statement.
[0005] Furthermore, merging the DataStream data stream and the newAddStream data stream to generate the DataSyncStream data stream includes the following steps: Merge the DataStream data stream and the newAddStream data stream; According to the real-time updated data synchronization information of the newAddStream data stream, the data synchronization logic and information are dynamically adjusted to generate the DataSyncStream data stream.
[0006] Furthermore, based on Flink, the DorisSink function is implemented. According to the data synchronization information, the DataSyncStream data stream is processed to generate the updated DataSyncStream data stream, which includes the following steps: Implement the DorisSink function based on Flink and determine whether the DataSyncStream data stream contains the Data attribute based on data synchronization information; If it does, check whether there is an OutputFormat corresponding to the source system table in outputFormatList. If not, create a new OutputFormat based on the existing table in Doris and add it to outputFormatList; If not, convert the data type to adapt to Doris; Generates the updated DataSyncStream data stream.
[0007] Furthermore, converting the data type to adapt to Doris includes the following steps: Determine whether the data synchronization information contains encrypted fields or custom fields. If so, add the isNewAdd property to the DataSyncStream data stream and set the value to 1. Then outputFormatList updates the OutputFormat corresponding to the source system table. Determine whether the data synchronization information removes the synchronization of a certain table. If so, add the isNewAdd property to the DataSyncStream data stream and set the value to 0, then outputFormatList removes the OutputFormat corresponding to the source system table.
[0008] Furthermore, writing the data in the updated DataSyncStream data stream into Doris includes the following steps; The data in outputFormatList is encapsulated as DorisDynamicOutputFormatJson object; Write the DorisDynamicOutputFormatJson object into Doris through the DorisStreamLoad method.
[0009] According to another embodiment of the present invention, a real-time dynamic collection system for multi-data center data to Doris based on FlinkCDC is provided, including: Data collection component: The data collection component collects the incremental log information of the binlog of the source system, encapsulates the incremental log information into a DataStream data stream, configures data synchronization information, and uses Flink to read the data synchronization information and generate a newAddStream data stream; Data processing component: The data processing component merges the DataStream data stream and the newAddStream data stream to generate the DataSyncStream data stream, and implements the DorisSink function based on Flink. According to the data synchronization information, the DataSyncStream data stream is processed to generate the updated DataSyncStream data stream; The data writing component writes the data in the updated DataSyncStream data stream into Doris.
[0010] Furthermore, the data collection component includes: The collection unit collects the incremental log information of the binlog of the source system and encapsulates the incremental log information into a DataStream data stream; Configuration unit: The configuration unit configures the source system connection information, table information, fields to be encrypted, and custom fields to be added required for synchronization. Flink is used to read data synchronization information and generate the newAddStream data stream.
[0011] Furthermore, the data writing component includes: Data encapsulation unit, the data encapsulation unit encapsulates the data in outputFormatList into a DorisDynamicOutputFormatJson object; StreamLoad control unit, the StreamLoad control unit writes the DorisDynamicOutputFormatJson object into Doris through the DorisStreamLoad method.
[0012] According to the embodiment of the present invention, a method and system for real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC is implemented, based on the Flink computing framework, and using the FlinkCDC component, real-time data collection, real-time computing, and consistency semantics are implemented; one-time configuration of data security encryption, custom field addition, and real-time playback of DDL changes is implemented through the data processing component; the Doris data synchronization component is optimized to realize the dynamic addition of DorisStreamLoad instances, thereby realizing the dynamic increase and decrease of synchronization tasks without stopping the service.
[0013] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a flowchart of a method for implementing real-time dynamic collection of multi-data center data to Doris based on FlinkCDC according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a real-time dynamic data collection system for multiple data centers to Doris based on FlinkCDC according to an embodiment of the present invention. DETAILED DESCRIPTION
[0015] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings to further illustrate the present invention.
[0016] First, combine Figure 1~2 A method and system for real-time dynamic collection of data from multiple data centers to Doris based on FlinkCDC according to an embodiment of the present invention is described, which is used for real-time dynamic collection of data from multiple data centers to Doris, and has a wide range of application scenarios.
[0017] like Figure 1As shown, the real-time dynamic collection method of multi-data center data to Doris based on FlinkCDC in the embodiment of the present invention includes the following steps: collecting binlog incremental log information of the source system, which can capture data changes in real time, ensure the real-time and dynamic nature of data synchronization, and encapsulate the incremental log information as a DataStream data stream. In this embodiment, there are three collection methods: the first is to collect the latest data; the second is to collect data after the specified offset; the third is a special case of the second, that is, when the offset is not specified, the system will collect according to the offset recorded in the latest log file, so that data collection and synchronization can be effectively performed in different situations; configure data synchronization information, use Flink to read the data synchronization information and generate a newAddStream data stream, and the configuration changes will be updated to the data stream in real time. In this embodiment, in the source system In the case of sharding, users only need to set appropriate regular expressions according to the sharding rules to achieve unified synchronization of relevant sharding data through one configuration; merge the DataStream data stream and the newAddStream data stream to generate the DataSyncStream data stream; implement the DorisSink function based on Flink, process the DataSyncStream data stream according to the data synchronization information, and generate an updated DataSyncStream data stream; write the data in the updated DataSyncStream data stream into Doris, and use Flink's powerful stream processing capabilities to merge and process different data streams to generate an updated data stream, and write the data to the target database Doris through the DorisSink function to ensure the efficiency and accuracy of data synchronization.
[0018] Further, if Figure 1 As shown, in this embodiment, configuring data synchronization information includes the following steps: configuring the source system connection information, table information, fields to be encrypted, and custom fields to be added required for synchronization. Configuring the fields to be encrypted can realize data encryption transmission between the source system and the target system, thereby enhancing data security; determining whether the Doris system has a target table corresponding to the source system table. If not, obtaining the table creation statement of the source system table and converting it into a Doris table creation statement. Based on the Doris-specific syntax parsing engine, the heterogeneous source system table structure can be automatically converted into a Doris table creation statement during synchronization; realizing automated table structure management and conversion, reducing the risk of errors in manual operations, and ensuring the consistency of table structures between different systems.
[0019] Further, if Figure 1As shown, in this embodiment, merging the DataStream data stream and the newAddStream data stream to generate the DataSyncStream data stream includes the following steps: merging the DataStream data stream and the newAddStream data stream; dynamically adjusting the data synchronization logic and information according to the data synchronization information updated in real time by the newAddStream data stream, generating the DataSyncStream data stream, maintaining the accuracy and real-time performance of data synchronization, and being flexible and efficient.
[0020] Further, if Figure 1 As shown, in this embodiment, the DorisSink function is implemented based on Flink, and the DataSyncStream data stream is processed according to the data synchronization information, and the updated DataSyncStream data stream is generated, which includes the following steps: based on Flink, the DorisSink function is implemented, and according to the data synchronization information, it is determined whether the DataSyncStream data stream contains the Data attribute; if it does, check whether there is an OutputFormat corresponding to the source system table in the outputFormatList, if not, a new OutputFormat is established according to the existing table in Doris and added to the outputFormatList; if it does not contain it, the data type is converted to adapt to Doris; the updated DataSyncStream data stream is generated. In this embodiment, if the DataStream data stream is a DML operation of the source system, the value of the main attribute data of the DataSyncStream data stream is the data collected by the DataStream data stream, and if the DataStream data stream is a DDL operation of the source system, the DataSyncStream data stream does not contain the Data attribute, and the DDL statement will immediately convert the data type to adapt to Doris through the AlterTableDruidFunction method, and then send it to Doris for execution, so as to ensure that the table structure of Doris is consistent with the source system. Through strict data type conversion and dynamic management of OutputFormat, the accuracy and consistency of data during transmission and storage are ensured.
[0021] Further, if Figure 1As shown, in this embodiment, converting the data type to adapt to Doris includes the following steps: determine whether the data synchronization information has encrypted fields or custom fields. If so, add the isNewAdd attribute to the DataSyncStream data stream and set the value to 1, then outputFormatList updates the OutputFormat corresponding to the source system table; determine whether the data synchronization information removes the synchronization of a certain table. If so, add the isNewAdd attribute to the DataSyncStream data stream and set the value to 0, then outputFormatList removes the OutputFormat corresponding to the source system table. In this embodiment, if there are encrypted fields and custom fields in the data synchronization information, the isNewAdd attribute is added to the DataSyncStream data stream and the value is set to 1; if the synchronization of a table is excluded in the data synchronization information, the isNewAdd attribute is added to the DataSyncStream data stream and the value is set to 0; when the DataSyncStream workflow contains the isNewAdd attribute and the attribute value is 1, the outputFormatList updates the OutputFormat corresponding to the source system table; when the DataSyncStream workflow contains the isNewAdd attribute and the attribute value is 0, the outputFormatList removes the OutputFormat corresponding to the source system table; when the DataSyncStream workflow does not contain the isNewAdd attribute and contains the data attribute, it is determined to be a DML operation. This method of dynamically adjusting data types and managing OutputFormat can effectively improve the flexibility, security and efficiency of the data synchronization process and adapt to the needs of complex data synchronization scenarios.
[0022] Further, if Figure 1 As shown, in this embodiment, writing the data in the updated DataSyncStream data stream into Doris includes the following steps: encapsulating the data in outputFormatList into a DorisDynamicOutputFormatJson object; writing the DorisDynamicOutputFormatJson object into Doris through the DorisStreamLoad method can improve data writing efficiency, enhance dynamic management capabilities, and have the advantages of real-time data synchronization. The StreamLoad instance already contains the synchronized table information when it is generated. When the DDL statement of the source system needs to modify the table information, the system will build a new StreamLoad to ensure data structure consistency, avoid synchronization interruptions, and achieve the purpose of real-time dynamic synchronization.
[0023] like Figure 2 As shown, according to another embodiment of the present invention, a real-time dynamic data collection system for multiple data centers to Doris based on FlinkCDC is provided, including: a data collection component, a data processing component and a data writing component.
[0024] Specifically, Figure 2 As shown, in this embodiment, the data acquisition component collects the binlog incremental log information of the source system, encapsulates the incremental log information into a DataStream data stream, configures data synchronization information, and uses Flink to read the data synchronization information and generate a newAddStream data stream; the data processing component merges the DataStream data stream and the newAddStream data stream to generate a DataSyncStream data stream, and implements the DorisSink function based on Flink, processes the DataSyncStream data stream according to the data synchronization information, and generates an updated DataSyncStream data stream; the data writing component writes the data in the updated DataSyncStream data stream into Doris.
[0025] Further, if Figure 2 As shown, in this embodiment, the data collection component includes: a collection unit, which collects the binlog incremental log information of the source system and encapsulates the incremental log information as a DataStream data stream; a configuration unit, which configures the source system connection information, table information, fields to be encrypted, and custom fields to be added required for synchronization, and uses Flink to read the data synchronization information and generate a newAddStream data stream.
[0026] Further, if Figure 2 As shown, in this embodiment, the data writing component includes: a data encapsulation unit, which encapsulates the data in outputFormatList into a DorisDynamicOutputFormatJson object; a StreamLoad control unit, which writes the DorisDynamicOutputFormatJson object into Doris through the DorisStreamLoad method.
[0027] Above, refer to Figure 1~2The present invention describes a method and system for implementing real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC according to an embodiment of the present invention. Based on the Flink computing framework, the FlinkCDC component is adopted to implement real-time data collection, real-time computing, and consistency semantics. The one-time configuration of data security encryption, custom field addition, and real-time playback of DDL changes is implemented through the data processing component. The Doris data synchronization component is optimized to implement the dynamic addition of DorisStreamLoad instances, thereby realizing the dynamic increase and decrease of synchronization tasks without interrupting services.
[0028] It should be noted that, in this specification, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprises..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0029] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.
Claims
1. A method for real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC, characterized in that: The following steps are included: Collect the incremental binlog log information of the source system and encapsulate the incremental log information into a DataStream data stream; Configure data synchronization information, use Flink to read the data synchronization information and generate a newAddStream data stream; Merge the DataStream data stream and the newAddStream data stream to generate a DataSyncStream data stream; Implement the DorisSink function based on Flink, process the DataSyncStream data stream according to the data synchronization information, and generate an updated DataSyncStream data stream; Write the data in the updated DataSyncStream data stream to Doris.
2. The method for implementing real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC as claimed in claim 1 is characterized in that: The configuration data synchronization information includes the following steps: Configure the source system connection information, table information, fields to be encrypted, and custom fields to be added required for synchronization; Determine whether the Doris system has a target table corresponding to the source system table. If not, obtain the table creation statement of the source system table and convert it into a Doris table creation statement.
3. The method for implementing real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC as claimed in claim 1 is characterized in that: The merging of the DataStream data stream and the newAddStream data stream to generate the DataSyncStream data stream comprises the following steps: Merge the DataStream data stream and the newAddStream data stream; According to the real-time updated data synchronization information of the newAddStream data stream, the data synchronization logic and information are dynamically adjusted to generate the DataSyncStream data stream.
4. The method for implementing real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC as claimed in claim 1 is characterized in that: The implementation of the DorisSink function based on Flink, processing the DataSyncStream data stream according to the data synchronization information, and generating an updated DataSyncStream data stream includes the following steps: Implement the DorisSink function based on Flink and determine whether the DataSyncStream data stream contains the Data attribute based on the data synchronization information; If it does, check whether there is an OutputFormat corresponding to the source system table in outputFormatList. If not, create a new OutputFormat based on the existing table in Doris and add it to outputFormatList; If not, convert the data type to adapt to Doris; Generates the updated DataSyncStream data stream.
5. The method for implementing real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC as described in claim 4 is characterized in that: The conversion of data types to adapt to Doris includes the following steps: Determine whether the data synchronization information contains encrypted fields or custom fields. If so, add the isNewAdd attribute to the DataSyncStream data stream and set the value to 1. Then, outputFormatList updates the OutputFormat corresponding to the source system table. Determine whether the data synchronization information removes the synchronization of a certain table. If so, add the isNewAdd attribute to the DataSyncStream data stream and set the value to 0, then outputFormatList removes the OutputFormat corresponding to the source system table.
6. The method for implementing real-time dynamic data collection from multiple data centers to Doris based on FlinkCDC as claimed in claim 5 is characterized in that: Writing the data in the updated DataSyncStream data stream to Doris includes the following steps: The data in outputFormatList is encapsulated as DorisDynamicOutputFormatJson object; Write the DorisDynamicOutputFormatJson object into Doris through the DorisStreamLoad method.
7. A real-time dynamic data collection system based on FlinkCDC to implement multi-data center data to Doris, characterized in that: Include: A data collection component collects the incremental log information of the binlog of the source system, encapsulates the incremental log information into a DataStream data stream, configures data synchronization information, and uses Flink to read the data synchronization information and generate a newAddStream data stream; A data processing component, which merges the DataStream data stream and the newAddStream data stream to generate a DataSyncStream data stream, and implements the DorisSink function based on Flink, processes the DataSyncStream data stream according to the data synchronization information, and generates an updated DataSyncStream data stream; A data writing component writes the data in the updated DataSyncStream data stream into Doris.
8. As claimed in claim 7, the multi-data center data is realized based on FlinkCDC to the Doris real-time dynamic collection system, characterized in that: The data acquisition component comprises: A collection unit, wherein the collection unit collects the incremental log information of the binlog of the source system and encapsulates the incremental log information into a DataStream data stream; A configuration unit, wherein the configuration unit configures the source system connection information, table information, fields to be encrypted, and custom fields to be added required for synchronization, and uses Flink to read the data synchronization information and generate a newAddStream data stream.
9. As claimed in claim 7, the multi-data center data is realized based on FlinkCDC to the Doris real-time dynamic collection system, characterized in that: The data writing component comprises: A data encapsulation unit, wherein the data encapsulation unit encapsulates the data in outputFormatList into a DorisDynamicOutputFormatJson object; A StreamLoad control unit, wherein the StreamLoad control unit writes the DorisDynamicOutputFormatJson object into Doris through the DorisStreamLoad method.
Citation Information
Cited By
Configurable big data import method and system based on Flink cdc
CN119248866A
