Data integration method and device based on cloud computing technology and computing equipment cluster

Abstract

The data integration method based on the cloud computing technology comprises the steps that a data reading end obtains change data in a source data source through a CDC assembly; the data reading end transmits the first identifier of each batch of changed data to the progress storage end, and transmits the changed data to the data writing end; the progress storage end records a first identifier from the data reading end; the data writing end writes the changed data into the target data source and transmits a second identifier of the data written into the target data source to the progress storage end; the progress storage end records a second identifier from the data writing end; and the data submitting end reads the first identifier and the second identifier recorded by the progress storage end, and submits the second identifier to the change data capturing assembly under the condition that the second identifier is consistent with any one of the first identifiers, so as to indicate the change data capturing assembly not to acquire data related to the second identifier from the source data source. According to the method, the data integration efficiency can be improved.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 21, 2023, with application number 202310745712.5 and application name “A CDC data integrated asynchronous high-performance read and write system”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of information technology (IT) technology, and in particular to a data integration method, device and computing device cluster based on cloud computing technology. Background Art

[0003] In the field of data integration, change data capture (CDC) refers to the acquisition of change data in real time by monitoring data changes in the data source, and then sending it to the downstream for business processing. The data sources supported by CDC include relational databases, non-relational databases, and data lakes. Data integration based on CDC can achieve real-time data synchronization between heterogeneous source data sources and target data sources. During the data integration process, data changes can be obtained from the source data source, and the data can be converted into the format required by the target data source, and finally written to the target data source. Among them, the end that obtains data changes from the source data source is called the data reading end; the end that converts the data and writes it to the target data source is called the data writing end.

[0004] In related technologies, such as Figure 1 As shown, in CDC data integration, a message queue (MQ) can be introduced as middleware to buffer the data obtained by the data reader. In the CDC data integration process, the data reader can read the changed data from the source data source through the CDC component, and transmit the changed data to the message queue through the MQ producer. At the same time, the MQ producer can commit the changed data it transmits to the CDC component (i.e., notify the CDC component which data has been read). The data writer can read the data that has not been written to the target data source from the message queue through the MQ consumer, convert the read data, and then call the database client to write the converted data to the target data source, and commit the data written to the target data source to the message queue (i.e., notify the message queue which data has been written). Figure 1Although the method shown can achieve data integration, it requires the introduction of third-party middleware to coordinate the data reading and writing ends, which increases the complexity of system construction and the development and operation costs. At the same time, since the data that needs to be written to the target data source needs to be stored in the third-party middleware, additional security management of user data is required, which is prone to data leakage. Summary of the invention

[0005] The present application provides a data integration method, apparatus, computing device cluster, computer storage medium and computer product based on cloud computing technology, which can reduce the cost of data integration system and improve data integration efficiency.

[0006] In a first aspect, the present application provides a data integration method based on cloud computing technology, which can be applied to a cloud computing platform. The cloud computing platform runs on an infrastructure, which includes multiple data centers set in different regions, and each data center includes multiple servers. The cloud computing platform includes: a data reading end, a data writing end, a progress storage end, and a data submission end. The method includes: the data reading end obtains at least one batch of change data in the source data source through a change data capture component, wherein the change data capture component is used to perceive the data change in the source data source and obtain the change data; the data reading end transmits the first identifier of each batch of data in at least one batch of change data to the progress storage end, and transmits at least one batch of change data to the data writing end; the progress storage end records the first identifier from the data reading end; the data writing end writes at least one batch of change data to the target data source, and transmits the second identifier of the data written to the target data source to the progress storage end; the progress storage end records the second identifier from the data writing end; the data submission end reads the first identifier and the second identifier recorded by the progress storage end, and, when the second identifier is consistent with any one of the first identifiers, submits the second identifier to the change data capture component to indicate that the change data capture component does not need to obtain data related to the second identifier from the source data source.

[0007] In this way, by separately recording the identifiers of the read data and the identifiers of the written data, and by comparing the identifiers of the read data with the identifiers of the written data, it is determined whether the read data has been written to the target data source, thereby achieving the purpose of coordinating data reading and writing, so that there is no need to wait for a batch of read data to be fully written before the next batch of data can be read, thereby improving data integration efficiency; at the same time, there is no need to set up additional third-party middleware to coordinate data reading and writing, thereby reducing the cost of the data integration system.

[0008] In a possible implementation, before the data reading end transmits the first identifier of each batch of data in at least one batch of changed data to the progress storage end, and transmits at least one batch of changed data to the data writing end, the method further includes: the data reading end obtains the third identifier of the data that has been written to the target data source from the progress storage end; and the data reading end filters the data in at least one batch of changed data based on the third identifier. In this way, the data that has been written to the target data source can be removed, thereby avoiding duplicate writing.

[0009] In a possible implementation, the data writing end writes at least one batch of changed data to the target data source, including: the data writing end aggregates the target batch of changed data into multiple sub-data based on the primary key or row index in the target batch of changed data, wherein the target batch of changed data is any batch of data in the at least one batch of changed data; the data writing end writes the multiple sub-data to the target data source in parallel, thereby improving the data writing efficiency.

[0010] In a possible implementation, there are multiple data writing ends. The data reading end transmits at least one batch of change data to the data writing end, including: the data reading end aggregates the target batch of change data into multiple sub-data based on the primary key or row index in the target batch of change data, and sends the multiple sub-data to different data writing ends respectively, wherein the target batch of change data is any batch of data in at least one batch of change data. The data writing end writes at least one batch of change data to the target data source, including: multiple data writing ends write multiple sub-data to the target data source in parallel. This improves the data writing efficiency.

[0011] In a possible implementation, before the data submission end submits the second identifier to the change data capture component, the method further includes: the data submission end obtains target information recorded by the progress storage end from the progress storage end, the target information is used to indicate the identifier that has been submitted to the change data capture component; and the data submission end determines, based on the target information, that the second identifier has not been submitted to the change data capture component, thereby avoiding duplicate submission.

[0012] In a possible implementation, after the data submission end submits the second identifier to the change data capture component, the method further includes: the data submission end transmits first information to the progress storage end, the first information being used to indicate that the second identifier has been submitted to the change data capture component; and the progress storage end records the first information. In this way, when the identifier is subsequently submitted to the CDC component, the identifier to be submitted can be filtered based on the recorded information indicating the identifier that has been submitted to the CDC component, thereby avoiding repeated submission.

[0013] In a possible implementation, the first identifier of the target batch of changed data is composed of at least a start offset and an end offset of the target batch of changed data in the source data source, wherein the target batch of changed data is any batch of data in the at least one batch of changed data.

[0014] In a second aspect, the present application provides a data integration device based on cloud computing technology, which is configured on a cloud computing platform. The cloud computing platform runs on an infrastructure, which includes multiple data centers set in different areas, and each data center includes multiple servers. The data integration device may include: a data reading module, a progress storage module, a data writing module, and a data submission module. The data reading module is used to read at least one batch of change data in a source data source through a change data capture component, wherein the change data capture component is used to sense the data change in the source data source and obtain the change data. The data reading module is also used to transmit the first identifier of each batch of data in at least one batch of change data to the progress storage module, and to transmit at least one batch of change data to the data writing module. The progress storage module is used to record the first identifier from the data reading module. The data writing module is used to write at least one batch of change data to a target data source, and to transmit the second identifier of the data written to the target data source to the progress storage module. The progress storage module is also used to record the second identifier from the data writing module. The data submission module is used to read the first identifier and the second identifier recorded by the progress storage module, and, when the second identifier is consistent with any one of the first identifiers, submit the second identifier to the change data capture component to indicate that the change data capture component does not need to obtain data related to the second identifier from the source data source.

[0015] In a possible implementation, before the data reading module transmits the first identifier of each batch of data in at least one batch of changed data to the progress storage module and transmits the at least one batch of changed data to the data writing module, the data reading module is further used to: obtain from the progress storage module the third identifier of the data that has been written to the target data source; and filter the data in the at least one batch of changed data based on the third identifier.

[0016] In a possible implementation, when the data writing module writes at least one batch of change data to the target data source, it is specifically used to: aggregate the target batch of change data into multiple sub-data based on the primary key or row index in the target batch of change data, wherein the target batch of change data is any batch of data in the at least one batch of change data; and write the multiple sub-data to the target data source in parallel.

[0017] In a possible implementation, there are multiple data writing modules. When the data reading module transmits at least one batch of change data to the data writing module, it is specifically used to: aggregate the target batch of change data into multiple sub-data based on the primary key or row index in the target batch of change data, and send the multiple sub-data to different data writing modules respectively, wherein the target batch of change data is any batch of data in the at least one batch of change data. When the data writing module writes at least one batch of change data to the target data source, it is specifically used to: multiple data writing modules write multiple sub-data to the target data source in parallel.

[0018] In a possible implementation, before submitting the second identifier to the CDC component, the data submission module is further used to: obtain target information recorded by the progress storage module from the progress storage module, where the target information is used to indicate the identifier that has been submitted to the change data capture component; and determine, based on the target information, that the second identifier has not been submitted to the change data capture component.

[0019] In a possible implementation, the data submission module is further used to: after the second identifier is submitted to the change data capture component, transmit first information to the progress storage module, the first information is used to indicate that the second identifier has been submitted to the change data capture component; the progress storage module is further used to: record the first information.

[0020] In a possible implementation, the first identifier of the target batch of changed data is composed of at least a start offset and an end offset of the target batch of changed data in the source data source, wherein the target batch of changed data is any batch of data in the at least one batch of changed data.

[0021] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0023] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device, causes the computing device to perform the method described in the first aspect or any possible implementation of the first aspect, or, when executed by a computing device cluster, causes the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0024] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the architecture of a data integration system provided in an embodiment of the present application;

[0026] Figure 2 It is a schematic diagram of the architecture of another data integration system provided in an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of the architecture of another data integration system provided in an embodiment of the present application;

[0028] Figure 4 It is a schematic diagram of the architecture of another data integration system provided in an embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of an interaction form between a tenant and a cloud computing platform provided in an embodiment of the present application;

[0030] Figure 6 It is a flow chart of a data integration method based on cloud computing technology provided in an embodiment of the present application;

[0031] Figure 7 It is a structural schematic diagram of a data integration device based on cloud computing technology provided in an embodiment of the present application;

[0032] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0033] Fig. 9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0034] Fig.10 It is a structural diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0036] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0037] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0038] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0039] For example, Figure 2 FIG. 2 shows another schematic diagram of the architecture of a data integration system provided by an embodiment of the present application. Figure 2 As shown, in this system integration system, the data reader can read a batch of changed data from the source data source through the CDC component and transmit it to the data writer. The data writer can convert the acquired data, then call the database client to write the converted data to the target data source, and commit the data written to the target data source to the CDC component in the data reader (i.e., notify the CDC component that the changed data read by the CDC component has been written to the target data source). Then, the data reader can read the next batch of changed data from the source data source through the CDC component. Figure 2The method shown does not require the introduction of third-party middleware, but each time the data reading end needs to wait for the data writing end to complete the batch writing before the next data reading operation can be performed. This causes the data writing end to block the data reading end, affecting the efficiency of data integration. If the data reading end waits for the data writing end to complete the batch writing before performing the next data reading operation, the data reading end may read the data that has been written to the target data source, which will result in duplicate writing in the future. Even if the data reading end records the data that has been read by itself, in the subsequent process of the data reading end, it is impossible to filter out the data that has not been written to the target data source from the data read by the data reading end, which will still result in duplicate writing.

[0040] In view of this, an embodiment of the present application provides another data integration system. In the data integration system, the identifier of the data read by the data reading end and the identifier of the data written by the data writing end can be recorded separately. By comparing the identifier of the data read by the data reading end and the identifier of the data written by the data writing end, it is determined whether the data read by the data reading end is written to the target data source. In this way, the data reading end can read the next batch of data without waiting for the data writing end to complete, thereby improving the efficiency of data integration; at the same time, there is no need to set up additional third-party middleware. Therefore, the data reading end and the data writing end are coordinated through the identifier of the data read by the data reading end and the identifier of the data written by the data writing end, thereby reducing the cost of the data integration system. In addition. In the data integration system, after reading the data, the data reading end can filter the data it reads by recording the identifier of the data written by the data writing end, thereby avoiding the situation of duplicate writing. The data integration system is introduced below.

[0041] For example, Figure 3 FIG. 2 shows a system architecture diagram of another data integration system provided by an embodiment of the present application. Figure 3 As shown, the system 300 of the data integration system includes: a source data source 310 , a target data source 320 , a data reading terminal 330 , a data writing terminal 340 , a progress storage terminal 350 and a CDC commit terminal 360 .

[0042] The source data source 310 is a terminal that provides data, and the target data source 320 is a terminal that writes data. Both the source data source 310 and the target data source 320 can be databases (such as cloud databases, etc.) or file systems, etc.

[0043] The data reading end 330 is an end for reading the changed data from the source data source 310. Exemplarily, the data reading end 330 can read the data generated by the change operations such as addition, update and deletion in the source data source 310. The data reading end 330 may include: a CDC component 331 and a data filtering component 332. The CDC component 331 may be used to sense the data changes in the source data source 310 and obtain the changed data. Exemplarily, the CDC component 331 may be, but is not limited to, Debezium, etc. The data filtering component 332 may be used to filter the changed data read by the CDC component 331 based on the identifier of the written data stored in the progress storage end 350 to remove the data written to the target data source 320. In addition, the data filtering component 332 may also transmit the read data obtained after filtering to the data writing end 340, and transmit the identifier of the read data obtained after filtering to the progress storage end 350. Exemplarily, the identifier of the read data may be composed of at least the start offset (start_offset) and the end offset (end_offset) of the read data in the source data source. The identifier of the written data is the same as the identifier of the read data, and may also be composed of at least the start offset (start_offset) and the end offset (end_offset) of the read data in the source data source. Exemplarily, the start_offset of a batch of data may be used to describe a starting point of the batch of data in the source data source; the start_offset of a batch of data may be used to describe the position offset of the end point of the batch of data in the source data source relative to the start_offset.

[0044] The data writing end 340 is an end for writing the data read by the data reading end 330 to the target data source 320. The data writing end 340 may include: a data conversion component 341 and a data writing component 342. The data conversion component 341 may be used to convert the format of the read data obtained from the data reading end 330 according to the data format of the data stored in the target data source 320, such as converting the field name or converting the field type. The data writing component 342 may be used to write the read data after the format conversion to the target data source 320, and transmit the identifier of the written data written to the target data source 320 to the progress storage end 350. In some embodiments, in order to improve the data writing performance, the data writing end 340 may aggregate the read data obtained according to the primary key or row index (Row_id) of the read data to obtain at least one batch of data. After aggregating multiple batches of data, the data writing end 340 may write these data to the target data source 320 in parallel through multiple threads.

[0045] The progress storage end 350 is an end for storing the identification of the read data, the identification of the written data and the identification of the read data that has been committed.

[0046] The CDC commit terminal 360 is a terminal that commits the identifier of the read data corresponding to the written data to the CDC component 331. The CDC commit terminal 360 can read the identifier of the read data and the identifier of the written data from the progress storage terminal 350 and compare the two. When the two are consistent, the CDC commit terminal 360 can commit the identifier of the read data to the CDC component 331, so that the CDC component 331 can know which data has been written to the target data source 320, so that the data will not be read later. In addition, the CDC commit terminal 360 can also transmit the identifier of the read data that has been committed to the progress storage terminal 350, so that the data recorded by the progress storage terminal 350 can be used to know which data has been committed and which has not been committed.

[0047] The above is an introduction to the data integration system 300 provided in the embodiment of the present application. In the above data integration system 300, there may be multiple data writing terminals 340. Figure 4 The data integration system 400 is shown. Figure 4 In the example, the data reader 330 can aggregate the read data it has obtained based on the primary key or row index (Row_id) of the read data to obtain multiple batches of data. Then, the data reader 330 can distribute different batches of data to different data writers 340. Figure 4 For other working processes in the data integration system 400 shown in FIG. Figure 3 The description of the data integration system 300 will not be repeated here.

[0048] In this embodiment, in the above-mentioned data integration systems 300 and 400, the data reading terminal 330, the data writing terminal 340, the progress storage terminal 350 and the CDC commit terminal 360 can all be configured on the cloud computing platform. Of course, the source data source 310 and the target data source 320 can also be configured on the cloud computing platform, which can be determined according to the actual situation and is not limited here. When the data reading terminal 330, the data writing terminal 340, the progress storage terminal 350 and the CDC commit terminal 360 are configured on the cloud computing platform, the tenant can purchase data integration services on the cloud computing platform to achieve data synchronization between the source data source and the target data source through the cloud computing platform. For example, Figure 5 FIG. 5 is a schematic diagram showing an interaction mode between a tenant and a cloud computing platform 500 provided in an embodiment of the present application. Figure 5As shown, the interaction between the tenant and the cloud computing platform 500 mainly includes: the tenant logs in to the cloud computing platform 500 through the client webpage, selects and purchases the cloud service of the data integration service in the cloud computing platform 500, and after the purchase, the tenant can synchronize the data between the source data source and the target data source based on the function provided by the data integration service. Among them, the cloud computing platform 500 can run on an infrastructure. The infrastructure may include multiple data centers set up in different regions, and each data center includes multiple servers. The data center may provide basic resources for the cloud computing platform 500, such as computing resources, storage resources, etc. Therefore, when the tenant purchases and uses the data integration service, the tenant mainly pays for the resources used. After using the data integration service, the tenant can complete the settings of the source data source, the target data source, and the data synchronization cycle through the configuration interface provided by the cloud computing platform 500 or the interface for interacting with the tenant. After the tenant completes the settings, the cloud computing platform 500 can synchronize the data between the source data source and the target data source according to the data synchronization cycle set by the tenant.

[0049] Of course, the data reading end 330, the data writing end 340, the progress storage end 350 and the CDC commit end 360 can also be configured on other nodes, and can be configured on the same node or on different nodes, which can be determined according to actual conditions and are not limited here. Among them, the node can be any device, server or device cluster with computing and processing capabilities.

[0050] Next, based on the above content, a data integration method based on cloud computing technology provided in an embodiment of the present application is introduced.

[0051] For example, Figure 6 The following is a flow chart of a data integration method based on cloud computing technology provided by an embodiment of the present application. Figure 5 The cloud computing platform described in the above. The cloud computing platform can be configured with a data reading terminal, a data writing terminal, a progress storage terminal and a data submission terminal. Figure 6 As shown, the data integration method based on cloud computing technology may include the following steps:

[0052] S601: The data reader reads at least one batch of changed data from a source data source through a CDC component.

[0053] In this embodiment, the data reader can read the change data in the source data source through the CDC component to obtain at least one batch of change data. Figure 3 The data reading end 330 in the CDC component can be the aforementioned Figure 3 The CDC component 331 in the source data source can be the aforementioned Figure 3 The source data source 310 in .

[0054] S602: The data reading end transmits the first identifier of each batch of data in at least one batch of changed data acquired by it to the progress storage end, and transmits the at least one batch of changed data to the data writing end.

[0055] In this embodiment, after the data reader obtains the change data, the first identifier of each batch of data in the change data can be transmitted to the progress storage end, so that the identifiers are recorded by the progress storage end. In some embodiments, the first identifier of a batch of change data can be at least composed of the starting offset (start_offset) and the ending offset (end_offset) of the batch of change data in the source data source. In addition, the data reader can also transmit the at least one batch of change data to the data writer end, so that the data can be written to the target data source through the data writer end. Exemplarily, the progress storage end can be the aforementioned Figure 3 The data storage end 350 in the data storage device may be the aforementioned Figure 3 The data is written to terminal 340.

[0056] S603: The progress storage end records the first identifier from the data reading end.

[0057] In this embodiment, after acquiring the first identifiers transmitted by the data reading end, the progress storage end may record these identifiers to store them.

[0058] S604: The data writing end writes at least one batch of changed data obtained from the data reading end to the target data source, and transmits the second identifier of the data written to the target data source to the progress storage end.

[0059] In this embodiment, after the data writing end obtains the changed data from the data reading end, the data writing end may write at least one batch of the changed data read to the target data source. Exemplarily, before writing the changed data to the target data source, the format of the changed data may be converted to the format of the data stored in the target data source. In addition, the data writing end may transmit the second identifiers of the data written to the target data source to the progress storage end, so as to record these identifiers through the progress storage end. Exemplarily, the target data source may be the aforementioned Figure 3 The target data source 332 in.

[0060] As a possible implementation, for writing any batch of data in at least one batch of read change data to the target data source, the data writing end may first aggregate the any batch of change data into multiple sub-data based on the primary key or row index in the any batch of change data. Then, the data writing end writes the multiple sub-data obtained by aggregation to the target data source in parallel, thereby improving the data writing efficiency.

[0061] S605. The progress storage end records the second identifier from the data writing end.

[0062] In this embodiment, after acquiring the second identifiers transmitted by the data writing end, the progress storage end may record these identifiers to store them.

[0063] S606. The data submission end reads the first identifier and the second identifier recorded by the progress storage end, and, when the second identifier is consistent with any one of the first identifiers, submits the second identifier to the change data capture component to indicate that the change data capture component does not need to obtain data related to the second identifier from the source data source.

[0064] In this embodiment, the data submission end can read the first identifier and the second identifier recorded by the progress storage end in real time or periodically (for example, every 1 second, every 1 minute, etc.), and compare the second identifier read by it with the first identifier. When the second identifier is consistent with any one of the first identifiers, it indicates that the change data associated with any one of the first identifiers has been written into the target data source. At this time, the data submission end can submit the second identifier to the CDC component to indicate that the CDC component does not need to obtain data related to the second identifier from the source data source. In this way, the subsequent CDC component will no longer read the change data associated with the second identifier. When there is no first identifier consistent with the second identifier, it indicates that all the change data associated with the first identifier have not been written into the target data source. Exemplarily, after submitting the second identifier to the CDC component, the data submission end can transmit information indicating that the second identifier has been submitted to the CDC component to the progress storage end; then, the progress storage end can record this information. In this way, before the data submission end subsequently submits the identifier to the CDC component, it can first obtain the information recorded in the progress storage end for indicating the identifier that has been submitted to the CDC component from the progress storage end; then, based on the read information, filter the identifier to be submitted to avoid repeated submission. In other words, before the data submission end submits the second identifier to the CDC component, it can first read the information recorded in the progress storage end for indicating the identifier that has been submitted to the CDC component from the progress storage end; then, based on the read information, determine whether the second identifier has not been submitted to the CDC component. When it is determined that the second identifier has not been submitted to the CDC component, the data submission end submits the second identifier to the CDC component. Exemplarily, the data submission end can be the aforementioned Figure 3 The CDC commit described in End 360.

[0065] In this way, by separately recording the identifiers of the read data and the identifiers of the written data, and by comparing the identifiers of the read data with the identifiers of the written data, it is determined whether the read data has been written to the target data source, thereby achieving the purpose of coordinating data reading and writing, so that there is no need to wait for a batch of read data to be fully written before the next batch of data can be read, thereby improving data integration efficiency; at the same time, there is no need to set up additional third-party middleware to coordinate data reading and writing, thereby reducing the cost of the data integration system.

[0066] In some embodiments, before the aforementioned S602, the data reader may also first obtain a third identifier of the data that has been written to the target data source from the progress storage end. Then, the data reader filters the data in the at least one batch of changed data read based on the third identifier to remove the data that has been written to the target data source, thereby avoiding duplicate writing.

[0067] In some embodiments, there may be multiple data writing ends. At this time, when the data reading end transmits data to the data writing end, it can aggregate the target batch of change data into multiple sub-data based on the primary key or row index in the target batch of change data, and send these multiple sub-data to different data writing ends respectively. Among them, the target batch of change data is any batch of data in at least one batch of change data obtained by the data reading end. Then, these multiple data writing ends can write the sub-data they each obtain to the target data source in parallel. Thereby improving the data writing efficiency.

[0068] It is understandable that the order of execution of the steps in the above embodiments does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The schemes in the above different embodiments can be combined according to actual conditions, and the combined schemes are still within the scope of this application.

[0069] The above is an introduction to the data integration method based on cloud computing technology provided by the embodiment of the present application. Next, based on the method in the above embodiment, a data integration device based on cloud computing technology provided by the embodiment of the present application is introduced.

[0070] For example, Figure 7 The schematic diagram of the structure of a data integration device based on cloud computing technology provided by an embodiment of the present application is shown. The device can be configured on a cloud computing platform. The cloud computing platform runs on an infrastructure, which includes multiple data centers located in different regions, each of which includes multiple servers such as Figure 7As shown, the data integration device 700 based on cloud computing technology may include: a data reading module 701, a progress storage module 702, a data writing module 703 and a data submission module 704.

[0071] The data reading module 701 is used to read at least one batch of changed data in the source data source through the changed data capture component, wherein the changed data capture component is used to sense the data changes in the source data source and obtain the changed data. The data reading module 701 is also used to transmit the first identifier of each batch of data in at least one batch of changed data to the progress storage module 702, and transmit at least one batch of changed data to the data writing module 703.

[0072] The progress storage module 702 is used to record the first identifier from the data reading module 701 .

[0073] The data writing module 703 is used to write at least one batch of changed data to the target data source, and transmit the second identifier of the data written to the target data source to the progress storage module 702 .

[0074] The progress storage module 702 is further used to record the second identifier from the data writing module 703 .

[0075] The data submission module 704 is used to read the first identifier and the second identifier recorded by the progress storage module 702, and, when the second identifier is consistent with any of the first identifiers, submit the second identifier to the change data capture component to indicate that the change data capture component does not need to obtain data related to the second identifier from the source data source. Figure 3 The data reading end 330 in the progress storage module 702 may be the aforementioned Figure 3 The progress storage terminal 350 in the data writing module 703 may be the aforementioned Figure 3 The data writing end 340 in the data submitting module 704 may be the aforementioned Figure 3 CDC commit end 360 in.

[0076] In some embodiments, before transmitting the first identifier of each batch of data in at least one batch of changed data to the progress storage module 702 and transmitting at least one batch of changed data to the data writing module 703, the data reading module 701 is also used to: obtain the third identifier of the data that has been written to the target data source from the progress storage module 702; and filter the data in at least one batch of changed data based on the third identifier.

[0077] In some embodiments, when the data writing module 703 writes at least one batch of change data to the target data source, it is specifically used to: aggregate the target batch of change data into multiple sub-data based on the primary key or row index in the target batch of change data, wherein the target batch of change data is any batch of data in the at least one batch of change data; and write multiple sub-data into the target data source in parallel.

[0078] In some embodiments, there are multiple data writing modules 703. When the data reading module 701 transmits at least one batch of change data to the data writing module 703, it is specifically used to: aggregate the target batch of change data into multiple sub-data based on the primary key or row index in the target batch of change data, and send the multiple sub-data to different data writing modules 703 respectively, wherein the target batch of change data is any batch of data in the at least one batch of change data. When the data writing module 703 writes at least one batch of change data to the target data source, it is specifically used to: multiple data writing modules 703 write multiple sub-data to the target data source in parallel.

[0079] In some embodiments, before submitting the second identifier to the CDC component, the data submission module 704 is further used to: obtain target information recorded by the progress storage module 702 from the progress storage module 702, where the target information is used to indicate the identifier that has been submitted to the change data capture component; based on the target information, determine that the second identifier has not been submitted to the change data capture component.

[0080] In some embodiments, the data submission module 704 is also used to: after the second identifier is submitted to the change data capture component, transmit the first information to the progress storage module 702, the first information is used to indicate that the second identifier has been submitted to the change data capture component; the progress storage module 702 is also used to: record the first information.

[0081] In some embodiments, the first identifier of the target batch of changed data is composed of at least a start offset and an end offset of the target batch of changed data in the source data source, wherein the target batch of changed data is any batch of data in the at least one batch of changed data.

[0082] In some embodiments, Figure 7 The data reading module 701, the progress storage module 702, the data writing module 703 and the data submitting module 704 shown in the figure can be implemented by software or by hardware. Exemplarily, the implementation of the data reading module 701 is described below by taking the data reading module 701 as an example. Similarly, the implementation of the progress storage module 702, the data writing module 703 and the data submitting module 704 can refer to the implementation of the data reading module 701.

[0083] As an example of a software functional unit, the data reading module 701 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the data reading module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0084] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0085] As an example of a hardware functional unit, the data reading module 701 may include at least one computing device, such as a server, etc. Alternatively, the data reading module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0086] The multiple computing devices included in the data reading module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the data reading module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data reading module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0087] It should be noted that, in other embodiments, the data reading module 701 can be used to execute any step of the data integration method based on cloud computing technology described in the above embodiments, and any one or more of the progress storage module 702, the data writing module 703 and the data submission module 704 can be used to execute any step of the data integration method based on cloud computing technology described in the above embodiments. In addition, the data reading module 701 can also be combined with any one or more of the progress storage module 702, the data writing module 703 and the data submission module 704 to be responsible for executing any step of the data integration method based on cloud computing technology described in the above embodiments. In addition, the steps that the data reading module 701, the progress storage module 702, the data writing module 703 and the data submission module 704 are responsible for implementing can also be specified as needed, and the data reading module 701, the progress storage module 702, the data writing module 703 and the data submission module 704 can respectively implement different steps of the data integration method based on cloud computing technology described in the above embodiments to achieve Figure 7 The data integration device 700 based on cloud computing technology has all the functions shown.

[0088] The present application also provides a computing device 800. Figure 8 As shown, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate through the bus 802. The computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0089] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8The bus 804 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 804 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 806, the processor 804, and the communication interface 808).

[0090] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0091] The memory 806 may include a volatile memory, such as a random access memory (RAM). The processor 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0092] The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the aforementioned Figure 7 The functions of the data reading module 701, the progress storage module 702, the data writing module 703 and the data submitting module 704 shown in the embodiment are implemented to realize the data integration method based on cloud computing technology described in the above embodiment. That is, the memory 806 stores instructions for executing the data integration method based on cloud computing technology described in the above embodiment.

[0093] Alternatively, the memory 806 stores executable codes, and the processor 804 executes the executable codes to respectively implement the aforementioned Figure 7 The functions of the data integration device 700 based on cloud computing technology shown in the figure are implemented to realize the data integration method based on cloud computing technology described in the above embodiment. That is, the memory 806 stores instructions for executing the data integration method based on cloud computing technology described in the above embodiment.

[0094] The communication interface 803 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0095] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0096] like Fig. 9 As shown, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the data integration method based on cloud computing technology described in the above embodiment.

[0097] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the data integration method based on cloud computing technology described in the above embodiment. In other words, the combination of one or more computing devices 800 can jointly execute instructions for executing the data integration method based on cloud computing technology described in the above embodiment.

[0098] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster may store different instructions, which are respectively used to execute the aforementioned Figure 7 The data integration device 700 based on cloud computing technology shown in FIG. 8 is a partial function. That is, the instructions stored in the memory 806 in different computing devices 800 can implement the functions of one or more modules in the data reading module 701, the progress storage module 702, the data writing module 703 and the data submission module 704.

[0099] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for executing the functions of the data reading module 701 and the progress storage module 702. At the same time, the memory 806 in the computing device 800B stores instructions for executing the functions of the data writing module 703 and the data submission module 704.

[0100] It should be understood that Fig.10The functions of the computing device 800A shown in FIG. 8 may also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B may also be completed by multiple computing devices 800.

[0101] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.10 The connection mode of the computing device cluster is different in that the memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the method in the above embodiment.

[0102] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store some instructions for executing the aforementioned data processing method. In other words, the combination of one or more computing devices 800 may jointly execute instructions for executing the aforementioned data processing method.

[0103] Based on the method in the above embodiment, the embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiment; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0104] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing device, enables the computing device to execute the method in the above embodiment, or, when executed by a computing device cluster, enables the computing device cluster to execute the method in the above embodiment.

[0105] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0106] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0107] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0108] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data integration method based on cloud computing technology, characterized in that: Applied to a cloud computing platform, the cloud computing platform runs on an infrastructure, the infrastructure includes multiple data centers arranged in different regions, each data center includes multiple servers, wherein the cloud computing platform includes: a data reading end, a data writing end, a progress storage end and a data submission end; The method comprises: The data reader obtains at least one batch of changed data in the source data source through a changed data capture component, wherein the changed data capture component is used to sense data changes in the source data source and obtain the changed data; The data reading end transmits the first identifier of each batch of data in the at least one batch of changed data to the progress storage end, and transmits the at least one batch of changed data to the data writing end; The progress storage end records the first identifier from the data reading end; The data writing end writes the at least one batch of changed data to the target data source, and transmits the second identifier of the data written to the target data source to the progress storage end; The progress storage end records the second identifier from the data writing end; The data submission end reads the first identifier and the second identifier recorded by the progress storage end, and, when the second identifier is consistent with any one of the first identifiers, submits the second identifier to the change data capture component to indicate that the change data capture component does not need to obtain data related to the second identifier from the source data source.

2. The method according to claim 1, characterized in that Before the data reading end transmits the first identifier of each batch of data in the at least one batch of changed data to the progress storage end, and before the at least one batch of changed data is transmitted to the data writing end, the method further includes: The data reading end obtains a third identifier of the data written to the target data source from the progress storage end; The data reader filters the data in the at least one batch of changed data based on the third identifier.

3. The method according to claim 1 or 2, characterized in that: The data writing end writes the at least one batch of changed data to the target data source, including: The data writing end aggregates the target batch of changed data into a plurality of sub-data based on a primary key or a row index in the target batch of changed data, wherein the target batch of changed data is any batch of data in the at least one batch of changed data; The data writing end writes the plurality of sub-data into the target data source in parallel.

4. The method according to claim 1 or 2, characterized in that: There are multiple data writing ends; The data reading end transmits the at least one batch of changed data to the data writing end, including: The data reading end aggregates the target batch of changed data into a plurality of sub-data based on a primary key or a row index in the target batch of changed data, and sends the plurality of sub-data to different data writing ends respectively, wherein the target batch of changed data is any batch of data in the at least one batch of changed data; The data writing end writes the at least one batch of changed data to the target data source, including: The plurality of data writing ends write the plurality of sub-data into the target data source in parallel.

5. The method according to any one of claims 1 to 4, characterized in that: Before the data submission end submits the second identifier to the changed data capture component, the method further includes: The data submission end obtains the target information recorded by the progress storage end from the progress storage end, wherein the target information is used to indicate an identifier submitted to the change data capture component; The data submission end determines, based on the target information, that the second identifier has not been submitted to the changed data capture component.

6. The method according to any one of claims 1 to 5, characterized in that: After the data submission end submits the second identifier to the changed data capture component, the method further includes: The data submission end transmits first information to the progress storage end, where the first information is used to indicate that the second identifier has been submitted to the change data capture component; The progress storage terminal records the first information.

7. The method according to any one of claims 1 to 6, characterized in that: The first identifier of the target batch of changed data is composed of at least a start offset and an end offset of the target batch of changed data in the source data source, wherein the target batch of changed data is any batch of data in the at least one batch of changed data.

8. A data integration device based on cloud computing technology, characterized in that: Configured on a cloud computing platform, the cloud computing platform runs on an infrastructure, the infrastructure includes a plurality of data centers arranged in different regions, each data center includes a plurality of servers; The data integration device comprises: A data reading module, used to read at least one batch of changed data in a source data source through a changed data capture component, wherein the changed data capture component is used to sense data changes in the source data source and obtain the changed data; The data reading module is further used to transmit the first identifier of each batch of data in the at least one batch of changed data to the progress storage module, and to transmit the at least one batch of changed data to the data writing module; A progress storage module, used for recording the first identifier from the data reading module; a data writing module, configured to write the at least one batch of changed data to a target data source, and transmit a second identifier of the data written to the target data source to the progress storage module; The progress storage module is further used to record the second identifier from the data writing module; A data submission module is used to read the first identifier and the second identifier recorded by the progress storage module, and, when the second identifier is consistent with any one of the first identifiers, submit the second identifier to the change data capture component to indicate that the change data capture component does not need to obtain data related to the second identifier from the source data source.

9. The device according to claim 8, characterized in that Before the data reading module transmits the first identifier of each batch of data in the at least one batch of changed data to the progress storage module and transmits the at least one batch of changed data to the data writing module, the data reading module is further used to: Acquire, from the progress storage module, a third identifier of the data that has been written to the target data source; Based on the third identifier, data in the at least one batch of changed data is filtered.

10. The device according to claim 8 or 9, characterized in that When the data writing module writes the at least one batch of changed data to the target data source, it is specifically used to: Aggregating the target batch of changed data into a plurality of sub-data based on a primary key or a row index in the target batch of changed data, wherein the target batch of changed data is any batch of data in the at least one batch of changed data; The plurality of sub-data are written to the target data source in parallel.

11. The device according to claim 8 or 9, characterized in that There are multiple data writing modules; When the data reading module transmits the at least one batch of changed data to the data writing module, the data reading module is specifically used to: Aggregating the target batch of changed data into a plurality of sub-data based on a primary key or a row index in the target batch of changed data, and sending the plurality of sub-data to different data writing modules respectively, wherein the target batch of changed data is any batch of data in the at least one batch of changed data; When the data writing module writes the at least one batch of changed data to the target data source, it is specifically used to: The plurality of data writing modules write the plurality of sub-data into the target data source in parallel.

12. The device according to any one of claims 8 to 11, characterized in that: Before submitting the second identifier to the CDC component, the data submission module is further used to: Acquire target information recorded by the progress storage module from the progress storage module, wherein the target information is used to indicate an identifier submitted to the change data capture component; Based on the target information, it is determined that the second identifier has not been submitted to the changed data capture component.

13. The device according to any one of claims 8 to 12, characterized in that: The data submission module is further used to: after the second identifier is submitted to the change data capture component, transmit first information to the progress storage module, wherein the first information is used to indicate that the second identifier has been submitted to the change data capture component; The progress storage module is further used to: record the first information.

14. The device according to any one of claims 8 to 13, characterized in that: The first identifier of the target batch of changed data is composed of at least a start offset and an end offset of the target batch of changed data in the source data source, wherein the target batch of changed data is any batch of data in the at least one batch of changed data.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-7.

16. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster comprises at least one computing device.

17. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster includes at least one computing device.

Images