Data synchronization methods, apparatus, devices and computer-readable storage media
By introducing data buffers between big data platforms for concurrent data reading, the issues of data inconsistency and reliability between big data platforms are resolved, achieving efficient and reliable data synchronization.
Patent Information
- Application Number
- CN202311716717.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-13
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-12-13
AI Technical Summary
In existing technologies, data synchronization between big data platforms suffers from time inconsistencies and low reliability, especially when cluster anomalies affect the data synchronization process.
By synchronizing the target data to the data buffer and triggering multiple big data platforms to concurrently read the data, synchronous storage is achieved, thus making the synchronization process of each major data platform independent.
It shortens the time of data inconsistency between big data platforms, improves the reliability of data synchronization, and ensures that the synchronization process of other platforms is not affected in case of abnormal situations.
Smart Images

Figure CN117688096B_ABST
Abstract
Description
Technical Field
[0001] This application pertains to big data processing technology, and particularly relates to a data synchronization method, apparatus, device, and computer-readable storage medium. Background Technology
[0002] Data is the core of a data platform, and data synchronization is the foundation for building a multi-center big data solution. Improving the timeliness and reliability of data synchronization between multiple big data platforms is one of the keys to building a multi-center big data cluster.
[0003] Typically, cross-cluster data synchronization is accomplished based on the data replication mechanism built into the big data cluster components. Data is written from the data source to the big data platform corresponding to one of the big data clusters, and then distributed to the big data platforms corresponding to other big data clusters through the data synchronization process within that big data cluster.
[0004] Thus, on the one hand, because data is written to various big data platforms in a sequential order, the time for data inconsistency between big data platforms is relatively long; on the other hand, because the cluster data synchronization configuration presents a multi-cluster chain structure, if an anomaly occurs in a cluster in the middle of the data synchronization link, it will affect the data synchronization process of other subsequent big data clusters, resulting in low reliability of data synchronization. Summary of the Invention
[0005] This application provides a data synchronization method, apparatus, device, and computer-readable storage medium, which can shorten the time of data inconsistency between big data platforms and improve the reliability of data synchronization.
[0006] In a first aspect, embodiments of this application provide a data synchronization method, the method comprising:
[0007] Synchronize the target data from the target data source to the data buffer;
[0008] Multiple big data platforms are triggered to concurrently read the target data from the data buffer, so that the multiple big data platforms can synchronously store the target data.
[0009] Secondly, embodiments of this application provide a data synchronization device, the device comprising:
[0010] The first synchronization module is used to synchronize the target data in the target data source to the data buffer.
[0011] The second synchronization module is used to trigger multiple big data platforms to concurrently read the target data from the data buffer, so that the multiple big data platforms can synchronously store the target data.
[0012] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions;
[0013] When the processor executes the computer program instructions, it implements the steps of the data synchronization method as described in any embodiment of the first aspect.
[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the data synchronization method as described in any embodiment of the first aspect.
[0015] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the steps of the data synchronization method as described in any embodiment of the first aspect.
[0016] The data synchronization method, apparatus, device, and computer-readable storage medium in this application embodiment synchronize target data from a target data source to a data buffer, and then trigger multiple big data platforms to concurrently read the target data from the data buffer, thereby enabling the multiple big data platforms to synchronously store the target data. Since each big data platform writes the target data synchronously, the time of data inconsistency between big data platforms can be shortened. Furthermore, since each big data platform reads the target data independently from the data buffer, the data synchronization processes of each big data platform are independent and do not affect each other. Therefore, even if one big data platform malfunctions, it will not affect the data synchronization process of other big data platforms, thereby improving the reliability of data synchronization. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a schematic diagram of an architecture applicable to the data synchronization method provided in this application.
[0019] Figure 2 This is a flowchart illustrating a data synchronization method provided in one embodiment of this application;
[0020] Figure 3 This is a data synchronization process architecture diagram provided in this application;
[0021] Figure 4This is a data access flowchart provided in this application;
[0022] Figure 5 This is a schematic diagram of the structure of a data synchronization device provided in one embodiment of this application;
[0023] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation
[0024] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0025] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0026] To address the problems of existing technologies, embodiments of this application provide a data synchronization method, apparatus, device, and computer-readable storage medium. This data synchronization method can be applied to cross-cluster data synchronization scenarios. The architecture to which the data synchronization method provided in this application is applicable is described below.
[0027] Figure 1 This is a schematic diagram of an architecture applicable to the data synchronization method provided in this application.
[0028] like Figure 1As shown, this architecture may include a target data source 11, a data buffer 12, and multiple big data platforms 13. The target data source 11 can be used to produce data, which may include batch data and streaming data. The data buffer 12 can be located between the target data source 11 and the big data platforms 13 to cache data to be stored, enabling the big data platforms 13 to concurrently and synchronously read data. The big data platforms 13 can be used to store data; one big data platform 13 can be a big data cluster, and the data stored in multiple big data platforms 13 can be the same. When new data needs to be stored, it can be synchronously stored in all the big data platforms 13.
[0029] Based on the above architecture, this application provides a data synchronization method. By synchronizing target data from a target data source to a data buffer, multiple big data platforms are triggered to concurrently read the target data from the data buffer, thereby enabling the multiple big data platforms to synchronously store the target data. This not only shortens the time of data inconsistency between big data platforms, but also ensures that even if one big data platform experiences an anomaly, it will not affect the data synchronization process of other big data platforms, thus improving the reliability of data synchronization.
[0030] The data synchronization method provided in the embodiments of this application will be described in detail below.
[0031] Figure 2 This is a schematic flowchart of a data synchronization method provided in one embodiment of this application. This data synchronization method can be executed by an electronic device such as a computer.
[0032] like Figure 2 As shown, this data synchronization method may specifically include the following steps:
[0033] S210. Synchronize the target data in the target data source to the data buffer;
[0034] S220: Trigger multiple big data platforms to concurrently read the target data from the data buffer, so that multiple big data platforms can synchronously store the target data.
[0035] Therefore, by synchronizing the target data from the target data source to the data buffer, and then triggering multiple big data platforms to concurrently read the target data from the data buffer, the multiple big data platforms can synchronously store the target data. Since each big data platform writes the target data synchronously, the time of data inconsistency between the big data platforms can be shortened. Furthermore, because each big data platform reads the target data independently from the data buffer, the data synchronization processes of each big data platform are independent and do not affect each other. Therefore, even if one big data platform experiences an anomaly, it will not affect the data synchronization process of other big data platforms, thereby improving the reliability of data synchronization.
[0036] The specific implementation methods for each of the above steps are described below.
[0037] In some implementations, in S210, the target data source can be any data source capable of producing data. The target data can be any data produced by the target data source; it can be batch data or streaming data. The data buffer can be a data storage system set up between the target data source and the big data platform for temporarily storing the target data to be stored; this data storage system can be a distributed data storage system.
[0038] For example, when there is target data that needs to be stored in the target data source, the target data can be first written to a data buffer for temporary storage, so that various big data platforms can subsequently read the target data from the data buffer. Considering the large data traffic generated when the big data platform synchronously reads data from the data buffer, a high-throughput storage component can be selected as the data buffer to ensure the performance of data synchronization. Furthermore, considering disaster recovery mechanisms, mutual backups can be designed for the data buffer as disaster recovery facilities in the event of data buffer anomalies.
[0039] In addition, different types of data buffers can be set up for different types of data to be stored. For example, a batch data buffer can be set up for batch data to be temporarily stored; a streaming data buffer can be set up for streaming data to be temporarily stored.
[0040] In some implementations, in S220, after the target data to be stored is written to the data buffer, multiple big data platforms that need to synchronize the target data can be triggered to concurrently read the target data from the data buffer. In this way, the multiple big data platforms can independently synchronize and store the target data.
[0041] Additionally, it should be noted that the methods to trigger multiple big data platforms to read data can include: multiple big data platforms subscribing to data from a data buffer, and once data is written to the data buffer, the multiple big data platforms that subscribed to the data can independently and concurrently read the data from the data buffer. Alternatively, the triggering method can also include: after data is written to the data buffer, a data read notification message is sent to multiple big data platforms, and upon receiving the read notification message, the multiple big data platforms can independently and concurrently read the data from the data buffer.
[0042] It should be emphasized that the embodiments of this application utilize the computing resources of each big data platform and adopt a data pull mode to concurrently and independently copy data from the data buffer and write it to the data cluster corresponding to the platform, rather than using a data push mode based on the computing resources of the data buffer to distribute a single piece of data to each big data platform simultaneously. Therefore, the embodiments of this application can save the computing resources of the centralized data buffer as much as possible and ensure that the data synchronization processes of each big data platform do not affect each other.
[0043] In some implementations, the data buffer may include a batch data buffer. Based on this, the above-described S210 may specifically include:
[0044] When the target data is batch data, the target data in the target data source is synchronized to the batch data buffer.
[0045] Here, the batch data can be, for example, data that is generated and stored periodically. The batch data buffer can be implemented, for example, using a high-throughput HDFS (Hadoop Distributed File System).
[0046] In addition, when writing batch data to the batch data buffer, necessary compression processing can be performed on the batch data to improve the transmission performance of massive data and save storage space in the batch data buffer.
[0047] For example, taking HDFS as a batch data buffer, HDFS can include a NameNode node for managing file system metadata and individual data nodes, and multiple horizontally scalable DataNode nodes for storing data. To ensure data reliability, HDFS uses a multi-replica mechanism, which enables data disaster recovery by storing multiple copies of the same data on multiple DataNode nodes. When writing target data to HDFS, HDFS can divide the data file corresponding to the target data into blocks and synchronously write them to multiple replicas in HDFS through a pipeline mechanism.
[0048] Therefore, in order to improve the data write performance when synchronizing target data to the batch data buffer and reduce the latency required to write target data to the batch data buffer, in some implementations, synchronizing target data from the target data source to the batch data buffer may specifically include:
[0049] The target data in the target data source is split into multiple data fragments.
[0050] The data writing tasks corresponding to multiple data shards are distributed to multiple nodes in the batch data buffer, so that multiple nodes can be used to write multiple data shards into the batch data buffer.
[0051] For example, taking data from a relational database as the target data, the data table corresponding to the target data can be sharded according to the hash value. The data writing task load corresponding to the multiple data shards can then be evenly distributed across multiple disks on multiple nodes in the batch data buffer. This allows the multiple nodes to concurrently write the multiple data shards into the batch data buffer. Each node can handle one or more data shard writing tasks, and the task allocation can be determined based on the node's load capacity.
[0052] In this way, by increasing the concurrency when writing target data to the batch data buffer, the throughput of the batch data buffer during data writing can be improved, and the latency required to write target data to the batch data buffer can be reduced.
[0053] In addition, to further improve the data write performance when synchronizing target data to the batch data buffer and reduce the latency required to write target data to the batch data buffer, in some embodiments, after synchronizing target data from the target data source to the batch data buffer, the data synchronization method provided in this application embodiment may further include:
[0054] Store a copy of the target data in the batch data buffer.
[0055] Here, regarding the multi-copy mechanism in related technologies that simultaneously writes multiple copies of data when writing target data to the batch data buffer, this application embodiment considers that the batch data buffer is only used as a temporary storage area for data and does not need to persistently store the written data. Therefore, in order to maximize the data writing performance, a portion of the data fault tolerance capability of the batch data buffer can be sacrificed. By reducing the number of copies set for the written data in the batch data buffer, for example, setting the number of copies to 1, the data writing performance of the batch data buffer can be maximized.
[0056] In this way, by storing only one copy of the target data in the batch data buffer, the time required to write the target data to the batch data buffer can be reduced, thereby further improving the data write performance when synchronizing the target data to the batch data buffer.
[0057] Based on this, for situations where the target data is batch data and the data buffer is a batch data buffer, in order to fully guarantee the timeliness of batch data synchronization, a "data-driven task scheduler" can be established. This task scheduler allows for the pre-setting of dependencies between data and tasks; that is, after the data files for each stage are generated, the task scheduler can initiate the next stage of data processing tasks. In some implementations, the above S220 may specifically include:
[0058] If it is determined that a first data file corresponding to the target data has been generated in the batch data buffer, the batch synchronization task that depends on the first data file is determined by the task scheduler.
[0059] The task scheduler triggers multiple big data platforms to execute batch synchronization tasks, enabling multiple big data platforms to concurrently read target data from the batch data buffer.
[0060] Here, the first data file can be the original data file corresponding to the target data generated in the batch data buffer after the target data is written into the batch data buffer.
[0061] For example, the dependency relationship between the original data file generated after writing batch data to the batch data buffer and the batch synchronization task used to read data from the batch data buffer can be set in advance in the task scheduler. Then, after the target data is written to the batch data buffer and a first data file corresponding to the target data is generated in the batch data buffer, the batch synchronization task dependent on the first data file can be determined by the task scheduler based on the pre-set dependency relationship. Thus, the task scheduler can drive each big data platform to execute the batch synchronization task, so that each big data platform can concurrently read the target data from the batch data buffer.
[0062] In addition, in some embodiments of this application, to synchronize data in the batch data buffer to various big data platforms, an external application encapsulation can be implemented based on the Distcp (Distributed Copy) capability built into the Hadoop component. Distcp can utilize the multi-node computing resources of the big data platform to synchronize data to the big data platform by initiating multi-process batch synchronization tasks. In the embodiments of this application, the data synchronization performance from the batch data buffer to the big data platform can be improved by increasing the concurrency of Distcp tasks.
[0063] Furthermore, in some embodiments, after S220 described above, the data synchronization method provided in this application embodiment may further include:
[0064] If it is determined that a second data file corresponding to the target data has been generated in the target big data platform, the file conversion task that depends on the second data file is determined by the task scheduler. The target big data platform can be any one of multiple big data platforms.
[0065] The task scheduler triggers the target big data platform to execute a file conversion task, so that the target big data platform can perform file conversion processing on the second data file to obtain the target type data file and store the target type data file.
[0066] Here, the second data file can be the original data file corresponding to the target data generated in the big data platform after the target data is written into the big data platform.
[0067] For example, the dependency relationship between the original data file generated after batch data is written to the big data platform and the file conversion task used to convert the original data file into a target type data file can be pre-set in the task scheduler. Then, after the target data is written to the big data platform and a second data file corresponding to the target data is generated in the big data platform, the task scheduler can determine the file conversion task dependent on the second data file based on the pre-set dependency relationship. This allows the task scheduler to drive the big data platform to execute the file conversion task, enabling the big data platform to convert the second data file corresponding to the target data into a target type data file and store it in the target type data file. The target type data file can be, for example, a columnar storage format suitable for batch task processing, such as ORC (Optimized Row Columnar) or Parquet, or a data file suitable for the underlying HBase data file format, such as HFile.
[0068] In addition, in some embodiments, after the target big data platform is triggered to execute a file conversion task via a task scheduler, the data synchronization method provided in this application may further include:
[0069] If a target type data file corresponding to the target data has been generated in the target big data platform, the task scheduler determines the file loading task that depends on the target type data file.
[0070] The task scheduler triggers the target big data platform to execute a file loading task, so that the target big data platform loads the target type data file into the corresponding big data component, and the big data component stores the target type data file.
[0071] Here, a big data platform can include various big data components, such as Hive, Impala, HBase, and HDFS. Different big data components can be used to store different types of target data.
[0072] For example, the task scheduler can be pre-configured to generate target type data files after batch data is written to the big data platform and converted into files, and to load the target type data files into the corresponding big data components. After the target data is written to the big data platform and the target type data files corresponding to the target data are generated in the big data platform, the task scheduler can determine the file loading tasks that depend on the target type data files based on the pre-configured dependencies. The task scheduler can then drive the big data platform to execute the file loading task, so that the big data platform loads the target type data files corresponding to the target data into the corresponding big data components, thereby completing the storage and backup of the target data.
[0073] In this way, by setting up the aforementioned data-driven task scheduler, data users can intuitively create dependencies on the required "data" rather than dependencies on "tasks." After the corresponding data file is generated, the data-driven task scheduler can initiate the next data processing task that depends on that data file, thus realizing data-driven task management. This avoids the data synchronization latency caused by the periodic scheduling and coarse-grained incremental change file detection mechanisms of existing big data components' storage layer synchronization, thereby fully ensuring data timeliness during batch data synchronization.
[0074] Furthermore, the aforementioned task scheduler can be further configured to meet the refined data storage requirements of multi-center big data platforms, enabling on-demand data storage across various big data platforms. For example, when synchronizing target data from the data buffer to the big data platform, data from different data sources, or even different data tables from the same data source, can be synchronized to different big data platforms for storage based on pre-configured data storage strategies.
[0075] Furthermore, in some implementations, the data buffer may also include a streaming data buffer. Based on this, the above-described S210 may specifically include:
[0076] When the target data is streaming data, the target data in the target data source is synchronized to the streaming data buffer.
[0077] Here, streaming data can be, for example, data generated and stored in real time. Since the synchronization of streaming data emphasizes the timeliness of data persistence, the streaming data buffer can be implemented using, for example, high-throughput, high-performance message middleware such as Kafka or Pulsar.
[0078] For example, when writing target data to the streaming data buffer, the target data can be partitioned for writing. Additionally, the data writing workload corresponding to data from different applications can be evenly distributed across multiple disks on multiple nodes to improve the throughput of the streaming data buffer during data writing. Simultaneously, the streaming data buffer can also be configured with a multi-node, multi-replica mechanism. That is, by storing multiple copies of the same data on multiple nodes, data disaster recovery can be achieved, and the reliability of data copy writing can be guaranteed based on the application's write replica ack configuration. Since streaming data emphasizes timeliness, and the operational costs of streaming data playback are higher than those of batch tasks, a message queue multi-replica mechanism can still be used in the streaming data buffer to ensure the disaster recovery capability of streaming data.
[0079] Furthermore, in cases where the target data is streaming data and the data buffer is a streaming data buffer, in some implementations, the above-mentioned S220 may specifically include:
[0080] Based on the data subscription mechanism, multiple big data platforms that subscribe to the target data concurrently read the target data from the streaming data buffer.
[0081] For example, each big data platform can independently subscribe to data in the message queue of the streaming data buffer. When data is stored in the message queue of the streaming data buffer, each big data platform can read the data it subscribed to and write the data into the big data component (such as HBase) that supports real-time data synchronization within the platform to achieve data synchronization.
[0082] It should be noted that, apart from the examples above, other processes for synchronizing streaming data can be similar to other processes for synchronizing batch data, and will not be elaborated here.
[0083] To better describe the overall solution, some specific examples are given based on the above embodiments.
[0084] For example, such as Figure 3The data synchronization process architecture diagram shown includes a data source system 31, a batch data buffer 32, a streaming data buffer 33, and multiple big data platforms, such as Big Data Platform A, Big Data Platform B, and Big Data Platform C. The data source system 31 can be the system where the target data source resides. The batch data buffer 32 can contain two mutually redundant batch data buffers, and the streaming data buffer 33 can also contain two mutually redundant streaming data buffers to improve the data buffer's disaster recovery capabilities.
[0085] In some examples, batch data to be stored read from data source system 31 can be synchronously written to batch data buffer 32. Each big data platform can independently and concurrently read the batch data to be stored from batch data buffer 32 for synchronous storage on each big data platform.
[0086] In other examples, streaming data to be stored read from data source system 31 can be synchronously written to streaming data buffer 33. Each big data platform can subscribe to data in the message queue of streaming data buffer 33. When the data it subscribes to is stored in streaming data buffer 33, the big data platform can independently and concurrently read the batch data to be stored from batch data buffer 32 for synchronous storage across the various big data platforms.
[0087] Furthermore, to shield external data access from the complexities of multi-center deployments and security authentication across multiple big data platforms, this application embodiment also establishes a collaborative service between the data access system and the big data platform. This service acts as a data routing and security authentication gateway, supporting high-availability switching in case of big data platform anomalies, while remaining transparent to external data access systems. Based on this, in some embodiments, after S220 above, the data synchronization method provided in this application embodiment may further include:
[0088] Receive the first data access request sent by the target data access system;
[0089] In response to the first data access request, the first big data platform to be accessed is determined from multiple big data platforms based on preset platform selection reference factors.
[0090] Obtain the first access address information corresponding to the first big data platform;
[0091] Send the first access address information to the target data access system so that the target data access system can access the first big data platform based on the first access address information.
[0092] Here, the target data access system can be any external data access system. The first access address information can be the address information used to access the first big data platform, such as the data access URL (Uniform Resource Locator) of the first big data platform.
[0093] For example, the target data access system can interface with a collaborative service to initiate a first data access request to a big data platform. This first data access request may include the path to the data file to be read or written. Upon receiving this first data access request, the collaborative service's internal "data routing" module can select the first big data platform from multiple big data platforms based on preset platform selection reference factors. The collaborative service can dynamically obtain the data access URL of the first big data platform by accessing its big data components (e.g., HDFS). At this point, the big data components in the first big data platform can dynamically allocate nodes as access nodes to respond to the data access request based on the cluster load within the platform. The collaborative service can then return the data access URL of the first big data platform to the target data access system. The target data access system can then access the selected first big data platform through this data access URL.
[0094] In some implementations, the platform may select reference factors such as at least one of query routing strategy, data storage readiness, and big data platform health.
[0095] For example, the collaborative service can first determine the main big data platform that is close to the target data access system from the multiple big data platforms according to the query routing strategy, as the candidate big data platform. Then, based on the data storage readiness status (e.g., whether the data has been synchronized) and the health status of the big data platform (e.g., whether the candidate big data platform has any abnormalities), the service can filter out the big data platform where the data is ready to be stored and the platform has no abnormalities, as the selected first big data platform.
[0096] In addition, to improve the security of the data access process, in some embodiments, the first data access request may include user authentication information. Accordingly, before obtaining the first access address information corresponding to the first big data platform, the data synchronization method provided in this application embodiment may further include:
[0097] The user authentication information is sent to the First Big Data Platform so that the First Big Data Platform can securely authenticate the user authentication information.
[0098] Receive security identification information returned by the First Big Data Platform when the user's authentication information is successfully verified.
[0099] Based on this, sending the first access address information to the target data access system as described above may specifically include:
[0100] Send security identification information and first access address information to the target data access system so that the target data access system can access the first big data platform based on the security identification information and first access address information.
[0101] Here, user authentication information may include, for example, a username and password used to verify the user's identity. Security identification information may be, for example, a security token issued by the First Big Data platform with a certain expiration time.
[0102] For example, before the collaborative service obtains the data access URL of the first big data platform, the "security gateway" module within the collaborative service can use user authentication information to connect to the selected first big data platform (Kerberos) for security authentication, and obtain a security token if the authentication is successful. The collaborative service can then return this security token along with the data access URL of the first big data platform to the target data access system, so that the target data access system can use the security token and access the selected first big data platform through the data access URL.
[0103] In addition, to achieve high-availability switching when the big data platform experiences an anomaly, in some implementations, after sending the first access address information to the target data access system, the data synchronization method provided in this application embodiment may further include:
[0104] Receive a second data access request sent by the target data access system. The second data access request is a data access request resent by the target data access system when it is unable to access the first big data platform.
[0105] In response to the second data access request, based on preset platform selection reference factors, the second big data platform to be accessed is re-determined from multiple big data platforms;
[0106] Obtain the second access address information corresponding to the second big data platform;
[0107] Send the second access address information to the target data access system so that the target data access system can access the second big data platform based on the second access address information.
[0108] Here, if the first big data platform malfunctions and becomes inaccessible, the collaborative service can switch the first big data platform to another accessible big data platform, such as the second big data platform, based on the health status and high availability strategy of each big data platform, so that the target data access system can continue to access the data normally.
[0109] For example, when the first big data platform is inaccessible, the target data access system can resend a data access request, i.e., a second data access request, to the collaborative service. Upon receiving this second data access request, the collaborative service can reselect a second, accessible big data platform from among multiple big data platforms based on preset platform selection reference factors. It then dynamically obtains the data access URL corresponding to the second big data platform by accessing its big data components. The collaborative service can return this data access URL to the target data access system, allowing the target data access system to access the second big data platform through this URL and continue data access on that platform.
[0110] In addition to configuring high availability policies for all systems or access interfaces, the high availability policies for collaborative services can also be configured at a fine-grained level, such as at the system level or interface level. This enables the distribution of data access load across big data clusters on different big data platforms and supports fine-grained high availability switching, flexibly responding to different cluster and data anomaly scenarios in production.
[0111] To better describe the overall solution, some specific examples are given based on the above embodiments.
[0112] For example, such as Figure 4 The data access flowchart shown can include S401-S407, which will be explained in detail below.
[0113] S401, The data access system initiates a data access request to the collaborative service.
[0114] S402. The collaborative service selects Big Data Platform B based on the query routing strategy, data storage readiness, and the health status of the Big Data Platform.
[0115] S403. The collaborative service uses user authentication information for security authentication and obtains a security token.
[0116] S404, Collaborative service obtains the data access URL of big data platform B.
[0117] S405, the collaborative service returns the obtained security token and the data access URL of the big data platform B to the data access system.
[0118] S406. The data access system uses a security token and the data access URL of the big data platform B to access data from the big data platform B.
[0119] S407. In the event of an anomaly in the big data platform B, switch to accessing the big data platform C through the high availability strategy of the collaborative service.
[0120] Therefore, by shielding external data access systems from the details of accessing the big data multi-center architecture, data table storage location, data access interfaces, and cluster security authentication during the data access process, this system supports the distribution of data access load across multiple big data platforms without degrading the access performance of the data access system to the big data platform. It ensures that the big data multi-center architecture and abnormal switching are transparent to external data access systems. Furthermore, it possesses fine-grained switching capabilities in abnormal big data platform scenarios, avoiding a unit-based "all-over" switch and minimizing the impact of switching a small number of abnormal data access services on other normally operating data access services.
[0121] It should be noted that the application scenarios described in the above embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0122] Based on the same inventive concept, this application also provides a data synchronization device. (Specifically combined with...) Figure 5 Please provide a detailed explanation.
[0123] Figure 5 This is a schematic diagram of the structure of a data synchronization device provided in one embodiment of this application.
[0124] like Figure 5 As shown, the data synchronization device 500 may include:
[0125] The first synchronization module 501 is used to synchronize the target data in the target data source to the data buffer.
[0126] The second synchronization module 502 is used to trigger multiple big data platforms to concurrently read the target data from the data buffer, so that the multiple big data platforms can synchronously store the target data.
[0127] The data synchronization device 500 described above will be explained in detail below:
[0128] In some embodiments, the data buffer includes a batch data buffer;
[0129] The first synchronization module 501 includes:
[0130] The first synchronization submodule is used to synchronize the target data in the target data source to the batch data buffer when the target data is batch data.
[0131] In some embodiments, the first synchronization submodule includes:
[0132] A data sharding unit is used to shard the target data in the target data source to obtain multiple data shards;
[0133] The task allocation unit is used to allocate the data writing tasks corresponding to the multiple data shards to multiple nodes in the batch data buffer, so as to use the multiple nodes to write the multiple data shards into the batch data buffer.
[0134] In some embodiments, the data synchronization device 500 further includes:
[0135] The replica storage module is used to store a replica of the target data in the batch data buffer after synchronizing the target data from the target data source to the batch data buffer.
[0136] In some embodiments, the second synchronization module 502 includes:
[0137] The first determining submodule is used to determine, through the task scheduler, batch synchronization tasks that depend on the first data file when it is determined that a first data file corresponding to the target data has been generated in the batch data buffer.
[0138] The first triggering submodule is used to trigger multiple big data platforms to execute the batch synchronization task through the task scheduler, so that the multiple big data platforms concurrently read the target data from the batch data buffer.
[0139] In some embodiments, the data synchronization device 500 further includes:
[0140] The first determining module is used to determine a file conversion task dependent on the second data file by a task scheduler after triggering multiple big data platforms to concurrently read the target data from the data buffer and determining that a second data file corresponding to the target data has been generated in the target big data platform. The target big data platform is any one of the multiple big data platforms.
[0141] The first triggering module is used to trigger the target big data platform to execute the file conversion task through the task scheduler, so that the target big data platform performs file conversion processing on the second data file to obtain a target type data file and stores the target type data file.
[0142] In some embodiments, the data synchronization device 500 further includes:
[0143] The second determining module is used to determine a file loading task that depends on the target type data file after the target big data platform is triggered to execute the file conversion task through the task scheduler, and after determining that a target type data file corresponding to the target data has been generated in the target big data platform;
[0144] The second triggering module is used to trigger the target big data platform to execute the file loading task through the task scheduler, so that the target big data platform loads the target type data file into the corresponding big data component and uses the big data component to store the target type data file.
[0145] In some embodiments, the data buffer includes a streaming data buffer;
[0146] The first synchronization module 501 includes:
[0147] The second synchronization submodule is used to synchronize the target data in the target data source to the streaming data buffer when the target data is streaming data.
[0148] In some embodiments, the second synchronization module 502 includes:
[0149] The second triggering submodule is used to trigger multiple big data platforms that subscribe to the target data to concurrently read the target data from the streaming data buffer based on the data subscription mechanism.
[0150] In some embodiments, the data synchronization device 500 further includes:
[0151] The first receiving module is configured to receive a first data access request sent by the target data access system after triggering multiple big data platforms to concurrently read the target data from the data buffer so that the multiple big data platforms can synchronously store the target data.
[0152] The first response module is used to respond to the first data access request by determining the first big data platform to be accessed from the plurality of big data platforms based on preset platform selection reference factors.
[0153] The first acquisition module is used to acquire the first access address information corresponding to the first big data platform;
[0154] The first sending module is used to send the first access address information to the target data access system, so that the target data access system can access the first big data platform according to the first access address information.
[0155] In some embodiments, the first data access request includes user authentication information;
[0156] The data synchronization device 500 also includes:
[0157] The information authentication module is used to send the user authentication information to the first big data platform before obtaining the first access address information corresponding to the first big data platform, so that the first big data platform can perform security authentication on the user authentication information.
[0158] The information receiving module is used to receive security identification information returned by the first big data platform when the user authentication information is successfully authenticated;
[0159] The first sending module is specifically used for:
[0160] The security identification information and the first access address information are sent to the target data access system so that the target data access system can access the first big data platform based on the security identification information and the first access address information.
[0161] In some embodiments, the data synchronization device 500 further includes:
[0162] The second receiving module is used to receive a second data access request sent by the target data access system after sending the first access address information to the target data access system. The second data access request is a data access request resent by the target data access system when it is unable to access the first big data platform.
[0163] The second response module is used to respond to the second data access request by re-determining the second big data platform to be accessed from the plurality of big data platforms based on preset platform selection reference factors.
[0164] The second acquisition module is used to acquire the second access address information corresponding to the second big data platform;
[0165] The second sending module is used to send the second access address information to the target data access system, so that the target data access system can access the second big data platform according to the second access address information.
[0166] In some embodiments, the platform selects reference factors including at least one of query routing strategy, data storage readiness, and big data platform health.
[0167] Therefore, by synchronizing the target data from the target data source to the data buffer, and then triggering multiple big data platforms to concurrently read the target data from the data buffer, the multiple big data platforms can synchronously store the target data. Since each big data platform writes the target data synchronously, the time of data inconsistency between the big data platforms can be shortened. Furthermore, because each big data platform reads the target data independently from the data buffer, the data synchronization processes of each big data platform are independent and do not affect each other. Therefore, even if one big data platform experiences an anomaly, it will not affect the data synchronization process of other big data platforms, thereby improving the reliability of data synchronization.
[0168] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application.
[0169] Electronic device 600 may include processor 601 and memory 602 storing computer program instructions.
[0170] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0171] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.
[0172] In certain embodiments, the memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.
[0173] The processor 601 implements any of the data synchronization methods described in the above embodiments by reading and executing computer program instructions stored in the memory 602.
[0174] In some examples, electronic device 600 may also include communication interface 603 and bus 610. For example, Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.
[0175] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0176] Bus 610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, bus 610 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0177] For example, the electronic device 600 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.
[0178] The electronic device 600 can execute the data synchronization method in the embodiments of this application, thereby achieving the combination Figure 1 and Figure 5 The data synchronization method and apparatus described.
[0179] Furthermore, in conjunction with the data synchronization methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data synchronization methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.
[0180] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0181] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0182] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0183] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.
[0184] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method of data synchronization, the method comprising: include: The target data in the target data source is synchronized to the data buffer; different types of data buffers are set for different data types to be stored; the data buffer includes a batch data buffer; when the target data is batch data, the target data in the target data source is synchronized to the batch data buffer. Multiple big data platforms are triggered to concurrently read the target data from the data buffer, so that the multiple big data platforms can synchronously store the target data; If it is determined that a second data file corresponding to the target data has been generated in the target big data platform, a file conversion task dependent on the second data file is determined by the task scheduler, wherein the target big data platform is any one of the plurality of big data platforms; The task scheduler triggers the target big data platform to execute the file conversion task, so that the target big data platform performs file conversion processing on the second data file to obtain a target type data file and stores the target type data file; If it is determined that a target type data file corresponding to the target data has been generated in the target big data platform, the task scheduler determines a file loading task that depends on the target type data file. The task scheduler triggers the target big data platform to execute the file loading task, so that the target big data platform loads the target type data file into the corresponding big data component, and uses the big data component to store the target type data file; The step of triggering multiple big data platforms to concurrently read the target data from the data buffer, so that the multiple big data platforms synchronously store the target data, includes: If it is determined that a first data file corresponding to the target data has been generated in the batch data buffer, a batch synchronization task dependent on the first data file is determined by the task scheduler, and multiple big data platforms are triggered by the task scheduler to execute the batch synchronization task, so that the multiple big data platforms use their respective computing resources to concurrently read the target data from the batch data buffer in a data pull mode; the task scheduler is pre-set with the dependency relationship between the data file and the next data processing task that depends on the data file.
2. The method of claim 1, wherein, The step of synchronizing the target data in the target data source to the batch data buffer includes: The target data in the target data source is split into multiple data fragments. The data writing tasks corresponding to the multiple data shards are assigned to multiple nodes in the batch data buffer, so that the multiple nodes can be used to write the multiple data shards into the batch data buffer.
3. The method of claim 1, wherein, After synchronizing the target data from the target data source to the batch data buffer, the method further includes: A copy of the target data is stored in the batch data buffer.
4. The method of claim 1, wherein, The data buffer includes a streaming data buffer; The step of synchronizing the target data from the target data source to the data buffer includes: If the target data is streaming data, the target data in the target data source is synchronized to the streaming data buffer.
5. The method of claim 4, wherein, The triggering of multiple big data platforms to concurrently read the target data from the data buffer includes: Based on the data subscription mechanism, multiple big data platforms that have subscribed to the target data are triggered to concurrently read the target data from the streaming data buffer.
6. The method of claim 1, wherein, After triggering multiple big data platforms to concurrently read the target data from the data buffer, so that the multiple big data platforms synchronously store the target data, the method further includes: Receive the first data access request sent by the target data access system; In response to the first data access request, based on preset platform selection reference factors, the first big data platform to be accessed is determined from the plurality of big data platforms; Obtain the first access address information corresponding to the first big data platform; The first access address information is sent to the target data access system so that the target data access system can access the first big data platform based on the first access address information.
7. The method of claim 6, wherein, The first data access request includes user authentication information; Before obtaining the first access address information corresponding to the first big data platform, the method further includes: The user authentication information is sent to the first big data platform so that the first big data platform can perform secure authentication on the user authentication information. Receive security identification information returned by the first big data platform when the user authentication information is successfully authenticated; Sending the first access address information to the target data access system includes: The security identification information and the first access address information are sent to the target data access system so that the target data access system can access the first big data platform based on the security identification information and the first access address information.
8. The method of claim 6, wherein, After sending the first access address information to the target data access system, the method further includes: Receive a second data access request sent by the target data access system, wherein the second data access request is a data access request resent by the target data access system when it is unable to access the first big data platform; In response to the second data access request, based on preset platform selection reference factors, the second big data platform to be accessed is re-determined from the plurality of big data platforms; Obtain the second access address information corresponding to the second big data platform; The second access address information is sent to the target data access system so that the target data access system can access the second big data platform based on the second access address information.
9. The method according to claim 6 or 8, characterized in that, The platform selects reference factors including at least one of the following: query routing strategy, data storage readiness, and the health status of the big data platform.
10. A data synchronization device, characterized in that, include: The first synchronization module is used to synchronize target data from the target data source to the data buffer; wherein, different types of data buffers are set for different data types to be stored; the data buffers include batch data buffers. The second synchronization module is used to trigger multiple big data platforms to concurrently read the target data from the data buffer, so that the multiple big data platforms can synchronously store the target data; The first determining module is used to determine, through a task scheduler, a file conversion task that depends on the second data file when a second data file corresponding to the target data has been generated in the target big data platform. The target big data platform is any one of the plurality of big data platforms. The first triggering module is used to trigger the target big data platform to execute the file conversion task through the task scheduler, so that the target big data platform performs file conversion processing on the second data file to obtain a target type data file and stores the target type data file; The second determining module is used to determine, through the task scheduler, a file loading task that depends on the target type data file when a target type data file corresponding to the target data has been generated in the target big data platform. The second triggering module is used to trigger the target big data platform to execute the file loading task through the task scheduler, so that the target big data platform loads the target type data file into the corresponding big data component and uses the big data component to store the target type data file; The first synchronization module includes a first synchronization submodule, which is used to synchronize the target data in the target data source to the batch data buffer when the target data is batch data; The second synchronization module includes a first determining submodule and a first triggering submodule; The first determining submodule is used to determine, through a task scheduler, a batch synchronization task that depends on the first data file when it is determined that a first data file corresponding to the target data has been generated in the batch data buffer; the task scheduler is pre-set with the dependency relationship between the data file and the next data processing task that depends on the data file. The first triggering submodule is used to trigger multiple big data platforms to execute the batch synchronization task through the task scheduler, so that the multiple big data platforms use their respective computing resources to concurrently read the target data from the batch data buffer in a data pull mode.
11. An electronic device, comprising: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the steps of the data synchronization method as described in any one of claims 1-9.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the steps of the data synchronization method as described in any one of claims 1-9.
Citation Information
Patent Citations
Multi-source heterogeneous incremental data synchronization method and system
CN111723160A
Data synchronization method and device, electronic equipment and readable storage medium
CN112115200A
Message queue-based data synchronization method, apparatus and device, and storage medium
CN112380227A
Big data cluster switching method and device, electronic equipment and storage medium
CN115757611A
