Real-time data synchronous recovery method and device based on database instance switching

Abstract

The invention provides a real-time data synchronous recovery method and device based on database instance switching. The method comprises the following steps: when a stream processing framework task starts, generating stream processing framework state mapping by configuring a state management mechanism in a framework and extracting full data. And then, based on the state mapping, when the flow processing framework task finds that the progress of the first instance is lost, performing second instance switching on the first instance to generate incremental synchronization data. And when the candidate instance logs are cleaned, performing snapshot acquisition and synchronization on the total data to generate total synchronous data. And finally, on the basis of a difference updating recovery algorithm, synchronizing the data, and finishing recovery of Flink real-time data synchronization. According to the embodiment of the invention, when the synchronous log progress is lost and the candidate instance is lost to change the log, the data can be ensured not to be lost, the complete reset in the acquisition stage is avoided, the automation and fault-tolerant capabilities are improved, and the manual intervention is reduced.

Description

Technical Field

[0001] The present application relates to the field of big data AI technology, and in particular to a real-time data synchronization recovery method and device based on switching database instances. Background Art

[0002] With the continuous development of big data technology and the increasing demand for real-time data processing, the stream processing framework combined with the change data capture technology (Flink-CDC) as an efficient data change capture technology is increasingly being used in modern data platforms. The core function of the stream processing framework combined with the change data capture technology is to capture and process the change data of the source database in the real-time stream processing system, thereby realizing the real-time transmission and processing of the data stream, which is widely used in scenarios such as log monitoring, data synchronization, and real-time analysis. Specifically, the stream processing framework combined with the change data capture technology uses the SourceEnumerator and SourceReader modules (both are components in Flink, used to allocate coordination partitions and read data from the allocated partitions respectively), and uses the metadata and state of the checkpoint to serialize and cache the coordinator metadata and the current state of the data shards, respectively, to ensure that during the data collection process, even if there are problems such as node failure or connection timeout, the task can be restored from the interruption point, and the Exactly-Once semantics are implemented to avoid repeated data processing.

[0003] However, in actual production, when the database change log is cleared prematurely or the log is lost due to other operation and maintenance operations (such as database expansion or migration), the Flink-CDC task will enter a suspended state and cannot be restored. This problem usually occurs when Flink-CDC cannot keep up with the speed of source change log generation in time, resulting in the clearing of the read sites and the inability to resume the transfer normally. The existing processing method mainly relies on manually restarting the task. However, this method is not only time-consuming and labor-intensive, but also affects downstream data consumption, generates duplicate data, and thus affects system performance and data consistency.

[0004] Therefore, one or more methods are needed to solve the above problems.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present application shows a real-time data synchronization recovery method and device based on switching database instances.

[0007] In a first aspect, the present application shows a real-time data synchronization recovery method based on switching database instances, the method comprising:

[0008] When the stream processing framework task starts, the state management mechanism of the stream processing framework is configured and the full amount of data is extracted to generate the stream processing framework state map;

[0009] Based on the stream processing framework state mapping, when the stream processing framework task finds that the progress of the first instance is lost, the incremental synchronization data is generated by switching the first instance to the second instance;

[0010] Based on the stream processing framework state mapping, when the candidate instance log is cleaned up, full synchronization data is generated by taking snapshots and synchronizing the full data;

[0011] Based on the differential update recovery algorithm, the real-time data synchronization in the stream processing framework is restored by synchronizing the incremental synchronization data and the full synchronization data.

[0012] In an exemplary embodiment of the present disclosure, the state management mechanism of the stream processing framework is configured, including:

[0013] When the stream processing framework task starts, a candidate instance is generated by constructing and configuring a data source connector, and full data is extracted from the candidate instance to generate full extracted data;

[0014] The candidate instances include a first instance and a second instance;

[0015] When the stream processing framework task starts, a field index is generated by constructing and configuring a data source connector;

[0016] Based on the field index, the state of the stream processing framework is mapped through the full amount of extracted data to generate a stream processing framework state map.

[0017] In an exemplary embodiment of the present disclosure, when the stream processing framework task finds that the progress of the first instance is lost, switching the first instance to the second instance includes:

[0018] Based on the stream processing framework state mapping, performing incremental data collection on the first instance through the stream processing framework task to generate first incremental data;

[0019] When the stream processing framework task finds that the progress of the first instance is lost, marking the preset earliest global transaction identifier of the first instance through the stream processing framework to generate a first mark;

[0020] Based on the loss of progress of the first instance, generating a second identifier range by checking a range of a global transaction identifier of the second instance;

[0021] When the second identifier range covers the first tag, the stream processing framework task switches to the second instance to collect incremental data and generate second incremental data;

[0022] When the second identifier range does not cover the first mark, the stream processing framework task reconnects to the first instance to synchronize the stream processing framework state log and generates an incremental synchronization log;

[0023] Based on the incremental synchronization log, incremental synchronization data is generated by synchronizing the first incremental data with the second incremental data.

[0024] In an exemplary embodiment of the present disclosure, when the candidate instance log is cleaned up, snapshot collection and synchronization of the full amount of data are performed, including:

[0025] When the candidate instance logs are cleaned up, the incremental data collection task of the candidate instance is switched to generate a full snapshot collection task;

[0026] Based on the full snapshot collection task, the data of the candidate instance is extracted in parallel in a field indexed sharded manner through the stream processing framework to generate full read data.

[0027] In an exemplary embodiment of the present disclosure, when the candidate instance log is cleaned up, snapshot collection and synchronization of the full amount of data are performed, including:

[0028] Based on the full read data, a full status log is generated by recording the status of the field index;

[0029] Based on the full status log, when the field index changes, full synchronization data is generated by comparing and synchronizing the full read data.

[0030] In an exemplary embodiment of the present disclosure, synchronizing the incremental synchronization data and the full synchronization data includes:

[0031] Based on the differential update recovery algorithm, the incremental synchronization data and the full synchronization data are synchronized to generate data synchronization information;

[0032] Based on the differential update recovery algorithm, the incremental synchronization log and the full status log are synchronized to generate log synchronization information;

[0033] Based on the data synchronization information and the log synchronization information, the data of the candidate instance is synchronously monitored to complete the restoration of real-time data synchronization in the stream processing framework.

[0034] In an exemplary embodiment of the present disclosure, it further includes:

[0035] Based on the data synchronization reliability principle, generating a candidate instance by introducing a backup data source into the candidate instance;

[0036] Based on the data synchronization reliability principle, an automatic alarm mechanism is generated by setting a monitoring alarm mechanism for the stream processing framework.

[0037] In a second aspect, the present application shows a real-time data synchronization recovery device based on switching database instances, the device comprising:

[0038] The state mapping module is used to configure the state management mechanism of the stream processing framework and extract the full amount of data when the stream processing framework task starts;

[0039] An incremental synchronization module, used for generating incremental synchronization data by switching the first instance to the second instance when the stream processing framework task finds that the progress of the first instance is lost;

[0040] The full synchronization module is used to generate full synchronization data by taking snapshots and synchronizing the full data when the candidate instance logs are cleaned up;

[0041] The real-time data synchronization module is used to restore the real-time data synchronization in the stream processing framework by synchronizing the incremental synchronization data and the full synchronization data.

[0042] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0043] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0044] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0045] The technical solution provided by this application may have the following beneficial effects:

[0046] Based on the embodiment of the present disclosure, when the stream processing framework task starts, the state management mechanism of the stream processing framework is configured, and the full amount of data is extracted to generate the stream processing framework state mapping. Afterwards, based on the stream processing framework state mapping, when the stream processing framework task finds that the progress of the first instance is lost, the incremental synchronization data is generated by switching the first instance to the second instance. And, when the candidate instance log is cleaned up, the full amount of synchronization data is generated by snapshot acquisition and synchronization of the full amount of data. Finally, based on the differential update recovery algorithm, the recovery of real-time data synchronization in the stream processing framework is completed by synchronizing the incremental synchronization data and the full amount of synchronization data. The embodiment of the present disclosure provides a multi-instance automatic switching and progress recovery solution based on an open source stream processing framework combined with change data capture technology. It can automatically switch the candidate instance and restore the synchronization log progress when the database instance fails or the progress is lost. When the synchronization log progress is lost and the candidate instance loses the change log, it can ensure that the data is not lost, avoid the complete reset of the acquisition stage, improve the automation and fault tolerance, and reduce manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of the steps of a real-time data synchronization recovery method based on switching database instances in the present application;

[0048] Figure 2 This is a system architecture diagram of a real-time data synchronization recovery method based on switching database instances in the present application;

[0049] Figure 3 It is a synchronization sample diagram of a candidate database instance of a real-time data synchronization recovery method based on switching database instances of the present application;

[0050] Figure 4 This is a flowchart of automatic instance switching of a real-time data synchronization recovery method based on switching database instances in the present application;

[0051] Figure 5 This is a differential update recovery and restart flow chart of a real-time data synchronization recovery method based on switching database instances in the present application;

[0052] Figure 6 It is a structural block diagram of a real-time data synchronization recovery device based on switching database instances of the present application;

[0053] Figure 7 is a block diagram of an electronic device of the present application;

[0054] Figure 8 is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] Flink: is an open source stream processing and batch processing framework for processing large-scale data streams and data batch processing tasks.

[0057] CDC (Change Data Capture): is a data management technology used to monitor and capture data changes in a database. It can detect additions, updates, and deletions in a database, capture these changes as events or logs, and then pass them to other systems or applications to reflect data changes in the database in real time or near real time.

[0058] Failover: A term used to describe the ability of a system or application to continue to operate normally in the event of a failure or abnormal situation. The goal of failover is to ensure the availability, reliability, and stability of the system to reduce the risk of service interruption and data loss.

[0059] Checkpoint: A mechanism for achieving fault tolerance and data consistency. It allows Flink applications to periodically create snapshots when processing streaming data so that they can recover from the previous state and continue processing data when a system failure occurs. Checkpoint helps ensure the accuracy and reliability of data processing while reducing the risk of data loss.

[0060] Barrier: used to implement global ordering of data streams and generation of watermarks, which are one of the key elements for event time processing and state consistency in distributed stream processing.

[0061] GTID (Global Transaction Identifier): is a mechanism used for global transaction identification in MySQL. It is a globally unique identifier that identifies each transaction in a MySQL replication cluster. GTID plays a key role in MySQL replication and high availability solutions, helping to ensure data consistency and fault tolerance.

[0062] Binlog (Binary Log): It is a binary log file in MySQL that records data change operations in the database. Binlog records write operations such as INSERT, UPDATE, and DELETE in the database, as well as data definition language (DDL) operations (for example, creating a table or changing the table structure). It is the transaction log of MySQL and is used to implement key functions such as data replication, recovery, and fault recovery.

[0063] Op (Operator): used in CDC to indicate the change log type, including operation enumeration values ​​such as (c)Create, (u)Update, (d)Delete, and (r)Read.

[0064] Reference Figure 1 , shows a flowchart of the steps of a real-time data synchronization recovery method based on switching database instances of the present application, which can be applied to electronic devices, wherein the method can specifically include the following steps:

[0065] Step S101, when the stream processing framework task starts, the state management mechanism of the stream processing framework is configured and the full amount of data is extracted to generate a stream processing framework state map;

[0066] Step S102, based on the stream processing framework state mapping, when the stream processing framework task finds that the progress of the first instance is lost, the first instance is switched to the second instance to generate incremental synchronization data;

[0067] Step S103, based on the stream processing framework state mapping, when the candidate instance log is cleaned up, full synchronization data is generated by performing snapshot collection and synchronization on the full data;

[0068] Step S104, based on the differential update recovery algorithm, the incremental synchronization data and the full synchronization data are synchronized to complete the recovery of the real-time data synchronization in the stream processing framework.

[0069] The disclosed embodiments ensure that data is not lost when the synchronization log progress is lost and the candidate instance loses the change log, avoids a complete reset of the collection phase, improves automation and fault tolerance, and reduces manual intervention.

[0070] Below, as Figure 2 As shown, a real-time data synchronization recovery method based on switching database instances in an embodiment of the present disclosure will be further explained.

[0071] Usage scenario: When extracting database instance change data logs in real time, a progress indicator appears or the file is cleared, requiring the entire database table to be re-extracted.

[0072] In step S101, when the stream processing framework task (the stream processing framework used in the present invention is: Flink, which is an open source stream processing and batch processing framework for processing large-scale data streams and data batch processing tasks) starts, the state management mechanism of the stream processing framework can be configured, and the full amount of data can be extracted to generate a stream processing framework state map.

[0073] In some optional embodiments of this example, Figure 3 As shown, taking database change data capture as an example, it is assumed that there are currently four database instances 192.168.0.1:7001, 192.168.0.2:7001, 192.168.0.3:7001, 192.168.0.4:7001 (subsequently simplified as the first instance, the second instance, the third instance, and the fourth instance), where the fourth instance is the master database, and the first instance, the second instance, and the third instance are slave databases.

[0074] When the stream processing framework task starts, add the two slave libraries 192.168.0.1:7001 and 192.168.0.2:7001 as candidate instances in the data source connector construction configuration (the priority is: the first instance is greater than the second instance).

[0075] Also, when configuring the data source connector, you need to configure the unique field index of the primary key + update time field.

[0076] Next, configure the state management mechanism of the stream processing framework, for example, use the map state (MapState, a state type in Flink, used to store, manage and map key-value pairs in stream computing) to store the primary key and update time of each record, and generate the state of the stream processing framework (fstate1). This state management will be used to compare whether there is any missing change data or duplicate data that needs to be processed each time the incremental log is extracted.

[0077] After the real-time task of the stream processing framework starts, the full phase is executed from the first instance according to the field index. Every time a row of data is read, the mapping relationship between the data and fstate1 is determined based on the primary key and update time.

[0078] For newly appeared records, the stream processing framework generates an op=c (Operator=create) change log. If the update time of the record is greater than the update time stored in fstate1, an op=u (Operator=update) change log is generated and the state fstate1 is updated. At the end of the collection task, if some records in the database do not appear in the state fstate1, these records will be marked as op=d (Operator=delete) and the corresponding change log will be generated.

[0079] In this phase, all data is read and stored in the state, generating a stream processing framework state map to ensure that the subsequent incremental phase can be processed accurately.

[0080] In step S102, based on the stream processing framework state mapping, when the stream processing framework task finds that the progress of the first instance is lost, incremental synchronization data is generated by switching the first instance to the second instance.

[0081] In some optional embodiments of this example, Figure 4 As shown in the figure, after executing the full synchronization phase, the incremental synchronization phase can be entered.

[0082] At this time, the first instance starts to collect incremental data and generate first incremental data. At the same time, the capacity expansion operation and maintenance operation needs to ensure that the use of the first instance can be queried without feeling through the active / standby switching.

[0083] However, because the database instance needs to re-obtain the mirror content from other master nodes when expanding capacity, binlog (Binary-Log is a binary log file in the database that records data change operations in the database. Binlog records input, update, deletion and other write operations in the database, as well as data definition language (DDL) operations (for example, creating a table or changing the table structure). Binlog is the transaction log of the database and is used to implement key functions such as data replication, recovery and fault recovery) only retains the records after obtaining the mirror. Therefore, after the primary-slave switch expansion, if there is still unobtained binlog in the first instance, the problem of progress loss will occur.

[0084] At this time, the stream processing framework acquisition task senses that the binlog progress is lost after the first instance is expanded, and marks the earliest global transaction identifier (GTID) record currently available in the first instance as the first marker, and attempts to switch to the next candidate instance: the second instance. Afterwards, compare whether the current available range of the global transaction identifier of the second instance can cover the range required for the current slice extraction (that is, the second identifier range covers the first marker). When the second identifier range covers the first marker, the binlog slice continues to be obtained in the second instance to generate the second incremental data. During this process, the stream processing framework maintains a current mapping state to record the progress and data range of the current switching instance.

[0085] Until the second identifier range does not cover the first mark, at this time, the stream processing framework generates an incremental synchronization log by reconnecting to the first instance and synchronizing the stream processing framework status log starting from the first mark, so as to achieve the synchronization of the first incremental data with the second incremental data and then generate incremental synchronization data to complete the synchronization of the data. At this stage, it should be noted that whether the first instance has synchronized the new binlog log to ensure that no data is lost during the recovery process. In addition, the stream processing framework needs to ensure that the progress of the first instance and the second instance is synchronized before returning to the normal incremental synchronization stage.

[0086] In step S103, based on the stream processing framework state mapping, when the candidate instance log is cleaned up, full synchronization data is generated by performing snapshot collection and synchronization on the full data.

[0087] In some optional embodiments of this example, suppose that after a period of time, the first instance and the second instance generate a large number of change logs in a short period of time. However, due to the limited resources of the stream processing framework extraction task, the consumption progress lags far behind the two slave libraries, and the first instance and the second instance retain limited logs. Under the file cleaning strategy, the binlogs that have not been synchronized by the extraction task are cleared, and the problem of progress loss occurs again.

[0088] In this case, the incremental synchronization of the stream processing framework will not be able to continue, the progress will be lost and cannot be recovered, so it is necessary to add a full snapshot collection task and switch to the full data extraction stage.

[0089] First, in order to improve the efficiency of full snapshot collection, the stream processing framework can extract data in parallel by sharding the data by primary key range. Each extraction task will be responsible for the data in a primary key range, thereby speeding up the reading process of the full data. Each time the data is read, it will be judged and stored according to the primary key + update time to generate full read data.

[0090] In some optional embodiments of this example, during the full data collection process, the stream processing framework will update fstate1 in real time based on the full read data to ensure that the processing status of each piece of data is correctly maintained. If the primary key and update time of a record already exist in the state and have not changed, the stream processing framework will skip the record to avoid generating redundant change logs. At the same time, data consistency issues may arise, especially when the primary key or update time changes.

[0091] After that, after the full data collection is completed, the stream processing framework will recalculate the current extracted data state and update the new data state to fstate1. Then, the stream processing framework will compare the new change log with the existing state, calculate the difference set, and generate the corresponding full state log.

[0092] Finally, the stream processing framework needs to synchronize and compare the full read data based on the primary key and update time to ensure the accuracy of the data. It then returns to the incremental synchronization stage to continue synchronizing data changes in real time.

[0093] In a specific example, if Figure 5 As shown, a full snapshot is taken of the first instance according to the priority. Assume that in a query slice (the slice content is select*from table where pk between a and b):

[0094] The first row of data X1, whose primary key pk1 does not exist in fstate1, generates a change log for X1 in the form of op=c (Operator=create operation) (such as {"db":"test","table":"order","key":{"order_id":"1001"},"after":{"order_id":"1001","pay_amount":56.6,"update_time":"2023-09-17 17:45:37.935488"},"op":"c"});

[0095] The second row of data X2, its primary key pk2 exists in fstate1, and the update time in X2 is greater than fstate1.get(pk2), then X2 is generated with op=u (Operator=update update operation) as the change log (such as {"db":"test","table":"order","key":{"order_id":"1002"},"before":{"order_id":"1002","pay_amount":16.2,"update_time":"2023-09-12 09:40:10.132463"},"after":{"order_id":"1002","pay_amount":30.0,"update_time":"2023-09-16 17:40:37.104376"},"op":"u"});

[0096] The third row of data X3, its primary key pk3 exists in fstate1, and the update time in X3 is equal to fstate1.get(pk3), so this record is ignored directly.

[0097] The primary keys pk1, pk2, and pk3 and their update times are recorded in another mapping state fstate2. After processing all the slices in the full stage, the difference between fstate1 and fstate2 is calculated. The primary key of the difference corresponds to the change log of op=d (Operator=delete operation) (such as {"db":"test","table":"order","key":{"order_id":"1003"},"after":{"order_id":"1003","pay_amount":0.6,"update_time":"2023-09-15 00:05:30.102301"},"op":"d"}).

[0098] After the full phase ends and the binlog generated in real time during the full phase is corrected, the incremental phase is finally re-entered.

[0099] In step S104, based on the differential update recovery algorithm, the incremental synchronization data and the full synchronization data are synchronized to complete the recovery of the real-time data synchronization in the stream processing framework.

[0100] In some optional embodiments of this example, based on the incremental synchronization data and the full synchronization data of the above steps, the stream processing framework will restore the synchronization of real-time data. The key is to read the incremental data from the latest binlog position of the first instance or the second instance and compare it with the current state. The stream processing framework will ensure that each data change can be correctly synchronized based on the data synchronization information of the primary key + update time stored in fstate1.

[0101] Also, after data synchronization, the stream processing framework needs to merge the change log generated in the full synchronization phase with the change log generated in the incremental synchronization phase. At this time, the stream processing framework will compare the log synchronization information based on the primary key and update time to ensure the consistency of the incremental synchronization log and the full status log to avoid duplicate processing.

[0102] In some optional embodiments of this example, in order to improve the reliability of data synchronization, it is preferred that a backup data source (such as a backup instance of the first instance or the second instance) can be introduced as a backup instance. When the main database fails or logs are lost, the stream processing framework can quickly switch to the backup instance for data synchronization, ensuring that data synchronization is not interrupted and reducing business impact.

[0103] At the same time, in order to promptly discover problems in data synchronization, it is preferred to set up a real-time monitoring and alarm mechanism. For example, CDC (Change Data Capture, which is a data management technology used to monitor and capture data changes in the database) can be used to monitor the binlog consumption progress, log cleaning, and the completion of the full snapshot. Once an abnormality occurs, the system will automatically alarm and trigger a recovery operation to ensure that the system can return to normal in the shortest time.

[0104] Also, in order to improve the fault tolerance and scalability of data synchronization, the state management of the stream processing framework can be stored in a distributed manner and combined with a highly available deployment solution.

[0105] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0106] Reference Figure 6 , shows a structural block diagram of a real-time data synchronization recovery device 400 based on switching database instances of the present application, the device comprising:

[0107] The state mapping module 401 is used to configure the state management mechanism of the stream processing framework and extract the full amount of data when the stream processing framework task starts;

[0108] The incremental synchronization module 402 is used to generate incremental synchronization data by switching the first instance to the second instance when the stream processing framework task finds that the progress of the first instance is lost;

[0109] The full synchronization module 403 is used to generate full synchronization data by performing snapshot collection and synchronization on the full data when the candidate instance log is cleared;

[0110] The real-time data synchronization module 404 is used to restore the real-time data synchronization in the stream processing framework by synchronizing the incremental synchronization data and the full synchronization data.

[0111] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0113] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] Figure 7 800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0115] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0116] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0117] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0118] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0119] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0120] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0121] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0122] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0123] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0124] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0125] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0126] Figure 8 19 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0127] Reference Figure 8 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0128] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0129] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0131] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A real-time data synchronization recovery method based on switching database instances, characterized in that: The method comprises: When the stream processing framework task starts, the state management mechanism of the stream processing framework is configured and the full amount of data is extracted to generate the stream processing framework state map; Based on the stream processing framework state mapping, when the stream processing framework task finds that the progress of the first instance is lost, the incremental synchronization data is generated by switching the first instance to the second instance; Based on the stream processing framework state mapping, when the candidate instance log is cleaned up, full synchronization data is generated by taking snapshots and synchronizing the full data; Based on the differential update recovery algorithm, the real-time data synchronization in the stream processing framework is restored by synchronizing the incremental synchronization data and the full synchronization data.

2. The method according to claim 1, characterized in that Configure the state management mechanism of the stream processing framework, including: When the stream processing framework task starts, a candidate instance is generated by constructing and configuring a data source connector, and full data is extracted from the candidate instance to generate full extracted data; The candidate instances include a first instance and a second instance; When the stream processing framework task starts, a field index is generated by constructing and configuring a data source connector; Based on the field index, the state of the stream processing framework is mapped through the full amount of extracted data to generate a stream processing framework state map.

3. The method according to claim 2, characterized in that When the stream processing framework task finds that the progress of the first instance is lost, the first instance is switched to the second instance, including: Based on the stream processing framework state mapping, performing incremental data collection on the first instance through the stream processing framework task to generate first incremental data; When the stream processing framework task finds that the progress of the first instance is lost, marking a preset earliest global transaction identifier of the first instance through the stream processing framework to generate a first mark; Based on the loss of progress of the first instance, generating a second identifier range by checking a range of a global transaction identifier of the second instance; When the second identifier range covers the first tag, the stream processing framework task switches to the second instance to collect incremental data and generate second incremental data; When the second identifier range does not cover the first mark, the stream processing framework task reconnects to the first instance to synchronize the stream processing framework state log and generates an incremental synchronization log; Based on the incremental synchronization log, incremental synchronization data is generated by synchronizing the first incremental data with the second incremental data.

4. The method according to claim 3, characterized in that When the candidate instance logs are cleaned up, snapshots of the full data are collected and synchronized, including: When the candidate instance logs are cleaned up, the incremental data collection task of the candidate instance is switched to generate a full snapshot collection task; Based on the full snapshot collection task, the data of the candidate instance is extracted in parallel in a field indexed sharded manner through the stream processing framework to generate full read data.

5. The method according to claim 4, characterized in that When the candidate instance logs are cleaned up, snapshots of the full data are collected and synchronized, including: Based on the full read data, a full status log is generated by recording the status of the field index; Based on the full status log, when the field index changes, full synchronization data is generated by comparing and synchronizing the full read data.

6. The method according to claim 5, characterized in that The incremental synchronization data and the full synchronization data are synchronized, including: Based on the differential update recovery algorithm, the incremental synchronization data and the full synchronization data are synchronized to generate data synchronization information; Based on the differential update recovery algorithm, the incremental synchronization log and the full status log are synchronized to generate log synchronization information; Based on the data synchronization information and the log synchronization information, the data of the candidate instance is synchronously monitored to complete the restoration of real-time data synchronization in the stream processing framework.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Based on the data synchronization reliability principle, generating a candidate instance by introducing a backup data source into the candidate instance; Based on the data synchronization reliability principle, an automatic alarm mechanism is generated by setting a monitoring alarm mechanism for the stream processing framework.

8. A real-time data synchronization recovery device based on switching database instances, characterized in that: include: The state mapping module is used to configure the state management mechanism of the stream processing framework and extract the full amount of data when the stream processing framework task starts; An incremental synchronization module, used for generating incremental synchronization data by switching the first instance to the second instance when the stream processing framework task finds that the progress of the first instance is lost; The full synchronization module is used to generate full synchronization data by taking snapshots and synchronizing the full data when the candidate instance logs are cleaned up; The real-time data synchronization module is used to restore the real-time data synchronization in the stream processing framework by synchronizing the incremental synchronization data and the full synchronization data.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images