Multi-region cloud platform data consistency processing method, device, product and system

Abstract

The present invention discloses a method, device, product and system for processing data consistency of a multi-region cloud platform, relating to the field of cloud computing. In this method, after detecting a change in the data of the central database, the data transformation is converted into a message in the message queue, and the message is transmitted across domains to the message queue of the edge-region cloud platform. The edge-region cloud platform updates its cache according to the message in the message queue. This ensures the data consistency between the database of the cloud platform deployed in the central region and the cache of the cloud platform deployed in the edge region. Moreover, after detecting a change in the data of the database of the cloud platform deployed in the central region, the data in the cache of the cloud platform deployed in the central region is updated, ensuring the data consistency between the cache of the cloud platform deployed in the central region and the database of the cloud platform deployed in the central region. That is, through this method, the data consistency among the data in the database, the data in the cache of the cloud platform deployed in the central region, and the cache of the cloud platform deployed in the edge region is achieved.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to a method, device, product and system for processing data consistency in a multi-region cloud platform. Background Art

[0002] With the continuous evolution of cloud platform services, there is a need for multi-region (Region) deployment. All data, including user information, resource information, etc., is maintained in the central region (Center Region), while only necessary functions are deployed in each edge region (Edge Region) to provide services. Data inconsistency in a multi-region cloud platform may lead to business errors, and in severe cases, may even cause system failures.

[0003] Related technologies for ensuring data consistency include the dual-write mode technology and the asynchronous cache update technology. In the dual-write mode technology, the database and the cache are written simultaneously. However, in a multi-region scenario, due to security considerations, the cache services in each region are not exposed externally. Therefore, the central cloud platform cannot directly write data changes to the cache of the edge cloud platform. In the asynchronous cache update technology, in a multi-region scenario, since the central database cannot connect to the message queue service in the edge region, the central database cannot directly notify the edge cloud platform. It can be seen that the related technologies for ensuring data consistency still cannot guarantee data consistency in a multi-region cloud platform, and the risks of business errors and system failures caused by data inconsistency still exist.

[0004] Therefore, how to ensure data consistency in a multi-region cloud platform is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device, product and system for processing data consistency in a multi-region cloud platform to solve the technical problem that the related technologies for ensuring data consistency cannot guarantee data consistency in a multi-region cloud platform.

[0006] To solve the above technical problem, the present invention provides a method for processing data consistency in a multi-region cloud platform, which is applied to a cloud platform deployed in a central region; the method includes:

[0007] Obtain a first data change event of a first database;

[0008] Process data in a first cache according to the first data change event, and store the first data change event in a first message queue; wherein, the first database, the first cache and the first message queue are all located in the cloud platform deployed in the central region;

[0009] Transmit the message in the first message queue that characterizes the first data change event across domains to the second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0010] On the one hand, before storing the first data change event in the first message queue, it further includes:

[0011] In the case where it is detected that the first data change event is a preset event, enter the step of storing the first data change event in the first message queue;

[0012] In the case where it is detected that the first data change event is not a preset event, process the first data change event locally.

[0013] On the other hand, the transmission of the message in the first message queue that characterizes the first data change event across domains to the second message queue in the cloud platform deployed in the edge region includes:

[0014] Configure the parameters of the message forwarding task in the plugin for message forwarding; wherein, the parameters of the message forwarding task at least include the first message queue, the second message queue, and the connection information between the first message queue and the second message queue;

[0015] Connect the plugin for message forwarding to the first message queue according to the parameters of the message forwarding task, pull the message that characterizes the first data change event from the first message queue, and transmit the message that characterizes the first data change event to the second message queue in the cloud platform deployed in the edge region.

[0016] On the other hand, the second message queue in the parameters of the message forwarding task is located in multiple cloud platforms deployed in the edge region;

[0017] The transmission of the message that characterizes the first data change event to the second message queue in the cloud platform deployed in the edge region includes:

[0018] Transmit the message that characterizes the first data change event to the second message queue in multiple cloud platforms deployed in the edge region.

[0019] On the other hand, the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, including:

[0020] The cloud platform deployed in the edge region parses the message in the second message queue; processes the data in the second cache in a single-threaded or serial manner and according to the parsed message.

[0021] After the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, it further includes:

[0022] Starting from when the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, within a preset duration, unprocessed message events are obtained and stored in the message queue for representing processing failures in the second message queue; the data in the second cache is processed according to the messages in the message queue for representing processing failures.

[0023] On the other hand, the cloud platform deployed in the edge region processing the data in the second cache according to the message in the second message queue includes:

[0024] The cloud platform deployed in the edge region obtains the priority order of the messages in the second message queue and the timestamps of receiving the messages;

[0025] Combining the priority order of the messages in the second message queue and the timestamp order of receiving the messages and processing the data in the second cache according to the content of the messages in the second message queue.

[0026] On the other hand, the method further includes:

[0027] The cloud platform deployed in the edge region receives the second data change event of the second database; wherein, the second database is located in the cloud platform deployed in the edge region;

[0028] Process the data in the second cache according to the second data change event.

[0029] On the other hand, there are multiple event listening and parsing resolvers for monitoring the first database in the cloud platform deployed in the central region, and there are multiple event listening and parsing resolvers for monitoring the second database in the cloud platform deployed in the edge region;

[0030] Obtaining the data change event of the database includes:

[0031] Obtain the data change event of the database through multiple event listening and parsing resolvers;

[0032] Processing the data in the cache according to the data change event includes:

[0033] Obtain the main event listening and parsing resolver from multiple event listening and parsing resolvers;

[0034] Process the data in the cache according to the data change event obtained by the main event listening and parsing resolver.

[0035] On the other hand, the event listening and parsing resolver obtaining the data change event of the database includes:

[0036] The event listening parser acts as a slave node of the database cluster and starts the input / output threads;

[0037] Initiate a data export request to the master node in the database cluster so that the master node in the database cluster starts a log export thread and uses the log to respond to the data export request of the event listening parser;

[0038] Receive and parse the log to obtain the data change events of the database.

[0039] On the other hand, the method further includes:

[0040] The cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database;

[0041] Empty the data in the second cache and store the data to be flushed into the second cache in the second cache;

[0042] Obtain the target information from the cloud platform deployed in the central region and store the target information in the second cache; wherein, the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region.

[0043] On the other hand, the method further includes:

[0044] Obtain the data to be flushed into the first cache from the first database;

[0045] Empty the data in the first cache and store the data to be flushed into the first cache in the first cache.

[0046] On the other hand, emptying the data in the second cache and storing the data to be flushed into the second cache in the second cache, or emptying the data in the first cache and storing the data to be flushed into the first cache includes:

[0047] When it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is a preset moment, or when it is detected that the user issues an instruction for triggering cache reset, empty the data in the second cache and store the data to be flushed into the second cache in the second cache, or empty the data in the first cache and store the data to be flushed into the first cache in the first cache.

[0048] On the other hand, after obtaining the data to be flushed into the first cache from the first database and before emptying the data in the first cache and storing the data to be flushed into the first cache in the first cache, it further includes:

[0049] Stop processing the data in the first cache according to the first data change event;

[0050] After emptying the data in the first cache and storing the data to be flushed into the first cache in the first cache, it further includes:

[0051] Process the data in the first cache according to the first data change event;

[0052] After the cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database, before emptying the data in the second cache and storing the data to be flushed into the second cache in the second cache, it further includes:

[0053] Stop processing the data in the second cache according to the second data change event;

[0054] After obtaining the target information from the cloud platform deployed in the central region and storing the target information in the second cache, it further includes:

[0055] Process the data in the second cache according to the second data change event.

[0056] On the other hand, after processing the data in the first cache according to the first data change event, it further includes:

[0057] Obtain the first data change event that has not been successfully processed;

[0058] Store the first data change event that has not been successfully processed in the message queue for representing processing failure in the first message queue;

[0059] Start an independent thread to process the first data change event that has not been successfully processed in the message queue for representing processing failure;

[0060] Delete the cache data corresponding to the first data change event that has not been successfully processed from the first cache;

[0061] After processing the data in the second cache according to the second data change event, it further includes:

[0062] Obtain the second data change event that has not been successfully processed;

[0063] Store the second data change event that has not been successfully processed in the message queue for representing processing failure in the second message queue;

[0064] Start an independent thread to process the second data change event that has not been successfully processed in the message queue for representing processing failure;

[0065] Delete the cached data corresponding to the second data change event that has not been successfully processed from the second cache.

[0066] On the other hand, the method further includes:

[0067] The cloud platform deployed in the edge region receives, through a unified interface, a request sent by a user for querying data from the second cache; if it is detected that the data corresponding to the request exists in the second cache, the data corresponding to the request is returned to the user; if it is detected that the data corresponding to the request does not exist in the second cache, the application programming interface of the cloud platform deployed in the central region is called to obtain the data corresponding to the request;

[0068] Or, receive, through a unified interface, a request sent by a user for querying data from the first cache;

[0069] Return the data corresponding to the request existing in the first cache to the user.

[0070] To solve the above technical problems, the present invention further provides a method for processing data consistency of a multi-region cloud platform, which is applied to a cloud platform deployed in an edge region. The method includes:

[0071] Receive the messages in the second message queue; wherein, the messages in the second message queue are obtained by the cloud platform deployed in the central region acquiring the first data change event of the first database; process the data in the first cache according to the first data change event, and store the first data change event in the first message queue; transmit the messages in the first message queue representing the first data change event across domains to the second message queue in the cloud platform deployed in the edge region; wherein, the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region;

[0072] Process the data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0073] To solve the above technical problems, the present invention further provides a device for processing data consistency of a multi-region cloud platform, which is applied to a cloud platform deployed in the central region; including:

[0074] An acquisition module, configured to acquire the first data change event of the first database;

[0075] A processing and storage module, configured to process the data in the first cache according to the first data change event, and store the first data change event in the first message queue; wherein, the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region;

[0076] A transmission module, configured to transmit, across domains, messages in the first message queue that are used to represent first data change events to a second message queue in a cloud platform deployed in an edge region, so that the cloud platform deployed in the edge region processes data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0077] To solve the above technical problems, the present invention further provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above multi-region cloud platform data consistency processing method are implemented.

[0078] To solve the above technical problems, the present invention further provides a multi-region cloud platform system, including:

[0079] A memory, configured to store a computer program;

[0080] A processor, configured to implement the steps of the above multi-region cloud platform data consistency processing method when executing the computer program.

[0081] To solve the above technical problems, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above multi-region cloud platform data consistency processing method are implemented.

[0082] The beneficial effects of the present invention are as follows. First, in this method, after detecting a change in the data of the central database, the data transformation is converted into a message in the message queue, and the message is transmitted across domains to the message queue of the cloud platform in the edge region. The cloud platform in the edge region updates its own cache according to the message in the message queue. This ensures the data consistency between the database of the cloud platform deployed in the central region and the cache of the cloud platform deployed in the edge region. Moreover, after detecting a change in the data of the database of the cloud platform deployed in the central region, the data in the cache of the cloud platform deployed in the central region is updated, which ensures the data consistency between the cache of the cloud platform deployed in the central region and the database of the cloud platform deployed in the central region. That is, through this method, the data consistency among the data in the database, the cache data of the cloud platform deployed in the central region, and the cache data of the cloud platform deployed in the edge region is achieved. Second, compared with the method of achieving data consistency by using the double-write mode, in a multi-region scenario, the present invention does not directly write data to the edge cache, but indirectly writes data to the edge cache through the message queue at the edge. Even if the cache services in each region are not exposed externally for security reasons, the solution of the present invention can still achieve data consistency among multi-region cloud platforms. Third, compared with the method of achieving data consistency by using asynchronous cache updates, in a multi-region scenario, the database of the cloud platform deployed in the central region of the present invention interacts with the message queue of the cloud platform deployed in the edge region through its own message queue, that is, the database of the cloud platform deployed in the central region is indirectly connected to the edge message queue through the central message queue, enabling the database of the cloud platform deployed in the central region to indirectly interact with the edge message queue and achieving data consistency among multi-region cloud platforms. In addition, a set of data located in the cloud platform deployed in the central region is shared among multiple regions. The cloud platform deployed in the edge region does not need to maintain a complete set of data and can obtain data consistent with the cloud platform deployed in the central region only through the cache and the message queue, reducing the deployment and maintenance costs. At the same time, better unified user management is achieved. Moreover, even when there is no data table in the database of the cloud platform deployed in the edge region, data consistency among multi-region cloud platforms is still achieved.

[0083] In addition, only when it is detected that the first data change event is a preset event, the first data change event is stored in the first message queue and then transmitted across domains; while when the first data change event is not a preset event, the first data change event is processed locally and no cross-domain transmission is required. Not all messages corresponding to the first data change events are transmitted across domains in this method, reducing the burden of irrelevant data transmission and improving the efficiency of achieving data consistency.

[0084] Configure the parameters of the message forwarding task in the plugin for message forwarding, and use the plugin for message forwarding to forward the message, without writing a custom cross-message queue bridge, and can be smoothly extended to more regions or multi-active data center scenarios. And the message is transmitted to the cloud platforms deployed in multiple edge regions through the plugin for message forwarding, realizing the message synchronization in multiple edge regions.

[0085] The cloud platforms deployed in the edge regions process the data in the second cache in a single-threaded or serial manner according to the parsed message, avoiding the risk of cache inconsistency caused by concurrent write operations. The cloud platforms deployed in the edge regions store the unprocessed message events in the second message queue, which is used to represent the message queue for processing failures; process the data in the second cache according to the messages in the message queue for processing failures. That is, through retry operations, the final consistency of the data is guaranteed.

[0086] The cloud platforms deployed in the edge regions are processed based on the priority or timestamp of the message, ensuring the real-time and accuracy of cache updates. By monitoring the database of the cloud platforms deployed in the edge regions and updating the cache of the cloud platforms deployed in the edge regions, it is ensured as much as possible that the data in the database of the cloud platforms deployed in the edge regions also exists in the cache.

[0087] There are multiple event listening parsers that monitor database changes in both the cloud platforms deployed in the central region and the cloud platforms deployed in the edge regions, increasing the probability of successfully capturing database change events; and when updating the cache, only update according to the events captured by the master node, avoiding the problem of data inconsistency between the database and the cache of the cloud platforms deployed in the central region caused by multiple event listening parsers updating the cache.

[0088] As a slave node of the database cluster, the event listening parser realizes the real-time monitoring of the log changes of the master node in the database cluster based on the master-slave protocol of the message queue, and realizes the monitoring of the data change events of the database.

[0089] The cloud platforms deployed in the edge regions obtain the data to be flushed into the second cache from the second database; clear the data in the second cache, and store the data to be flushed into the second cache in the second cache; obtain the target information from the cloud platforms deployed in the central region and store the target information in the second cache; where the target information is located in the cloud platforms deployed in the central region and not in the cloud platforms deployed in the edge regions. It realizes the full refresh of the data of the cloud platforms deployed in the edge regions, can correct the incremental updates that the cloud platforms deployed in the edge regions may miss during offline or network anomalies, and keeps the cache consistent with the data of the cloud platforms deployed in the central region finally.

[0090] The cloud platform deployed in the central region obtains the data to be written into the first cache from the first database; clears the data in the first cache, and stores the data to be written into the first cache in the first cache. The full refresh of the data of the cloud platform deployed in the central region is realized, and the incremental updates that may be missed by the cloud platform deployed in the central region during offline or network anomalies can be corrected, so that the cache of the cloud platform deployed in the central region is finally consistent with the database of the cloud platform deployed in the central region.

[0091] When it is detected that the multi-region cloud platform is started, or when it is detected that the current time is a preset time, or when it is detected that the user issues an instruction indicating the trigger to reset the cache, the full refresh of the data of the cloud platform deployed in the central region and the full refresh of the data of the cloud platform deployed in the edge region are performed, ensuring data consistency; and compared with the method of performing full refresh in all scenarios, the resource consumption is reduced.

[0092] When performing a full refresh, stop processing the data in the cache according to the data change event, that is, stop the real-time cache update operation; after the full refresh is completed, continue the real-time cache update operation to ensure the integrity of the cache data.

[0093] For the data change events that are not processed successfully, store them in the message queue indicating processing failure, start an independent thread for processing, and delete the corresponding data from the cache to ensure the final data consistency.

[0094] Set a unified interface query in both the cloud platform deployed in the central region and the cloud platform deployed in the edge region. The cloud platform deployed in the edge region or other callers only need to access the unified interface, avoiding frequent cross-region access to the database of the cloud platform deployed in the central region, improving the query performance and simplifying the system design.

[0095] The present invention also provides a multi-region cloud platform data consistency processing method applied to the cloud platform deployed in the edge region, a multi-region cloud platform data consistency processing device, a computer program product, a multi-region cloud platform, and a computer-readable storage medium, which have the same or corresponding technical features as the multi-region cloud platform data consistency processing method applied to the cloud platform deployed in the central region described above, and the effects are the same. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0097] Figure 1Schematic diagram of a multi-region cloud platform provided by the present invention;

[0098] Figure 2 Flowchart of a method for processing data consistency in a multi-region cloud platform provided by the present invention;

[0099] Figure 3 Schematic diagram of the working principle of an event listening parser provided by an embodiment of the present invention;

[0100] Figure 4 Flowchart of a data transmission method provided by an embodiment of the present invention;

[0101] Figure 5 Overall schematic diagram of a data consistency method for a multi-region deployed cloud platform provided by an embodiment of the present invention;

[0102] Figure 6 Structural diagram of a multi-region cloud platform system provided by an embodiment of the present invention. Detailed implementation manners

[0103] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0104] The core of the present invention is to provide a method, device, product and system for processing data consistency in a multi-region cloud platform to solve the technical problem that the related technologies for ensuring data consistency cannot ensure the data consistency of a multi-region cloud platform.

[0105] Figure 1 Schematic diagram of a multi-region cloud platform provided by the present invention, as Figure 1 shown, including a cloud platform deployed in the central region and a cloud platform deployed in the edge region. Both the cloud platform deployed in the central region and the cloud platform deployed in the edge region include their own databases and caches.

[0106] The cloud platform deployed in the central region maintains all data, including user information, resource information, etc., while the cloud platforms deployed in each edge region only deploy necessary functions to provide services. Since user information is only stored in the database of the cloud platform deployed in the central region and there is no corresponding user record in the database of the cloud platform deployed in the edge region, therefore, in the present invention, data consistency of the multi-region cloud platform needs to ensure data consistency among the database of the cloud platform deployed in the central region, the cache of the cloud platform deployed in the central region, and the cache of the cloud platform deployed in the edge region.

[0107] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. A method for processing data consistency in a multi-region cloud platform provided by the present invention is applied to a cloud platform deployed in a central region. Figure 2 The flowchart of a method for processing data consistency in a multi-region cloud platform provided by the present invention is as Figure 2 shown, and the method includes:

[0108] S10: Obtain a first data change event of a first database; wherein, the first database is located in the cloud platform deployed in the central region;

[0109] S11: Process the data in the first cache according to the first data change event, and store the first data change event in a first message queue; wherein, both the first cache and the first message queue are located in the cloud platform deployed in the central region;

[0110] S12: Transmit the message in the first message queue representing the first data change event across domains to a second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0111] The changes in the database include adding, deleting, or updating data. There is no limitation on the method for obtaining the first data change event of the first database, as long as the changes in the database can be monitored. In the present invention, an event listening and parsing device for monitoring the first database is deployed in the cloud platform deployed in the central region. In order to improve the probability of successfully capturing database change events, in practice, multiple event listening and parsing devices for monitoring the first database are deployed in the cloud platform deployed in the central region (i.e., multi-copy deployment). Obtaining the data change event of the database includes: obtaining the data change event of the database through multiple event listening and parsing devices. In addition, in order to avoid the problem of data inconsistency between the database and the cache in the cloud platform deployed in the central region caused by multiple event listening and parsing devices all updating the cache, processing the data in the cache according to the data change event includes: obtaining the main event listening and parsing device from multiple event listening and parsing devices; processing the data in the cache according to the data change event obtained by the main event listening and parsing device. There is no limitation on the selected main event listening and parsing device, which is determined according to the actual situation.

[0112] After obtaining the first data change event, if all the first data change events are transmitted to the cloud platform deployed in the edge region, the amount of data to be transmitted is relatively large. Therefore, before storing the first data change event in the first message queue, it further includes:

[0113] When it is detected that the first data change event is a preset event, enter the step of storing the first data change event into the first message queue;

[0114] When it is detected that the first data change event is not a preset event, process the first data change event locally.

[0115] There is no limitation on the preset event. For example, in the database of the cloud platform deployed in the edge area, user data is not stored. Therefore, user data can be obtained from the cloud platform deployed in the central area. At this time, the preset event is an event related to users.

[0116] The data change events obtained by the event listening parser from the database include:

[0117] The event listening parser acts as a slave node of the database cluster and starts the input / output thread;

[0118] Initiate a data export request to the master node in the database cluster so that the master node in the database cluster starts a log export thread and uses the log to respond to the data export request of the event listening parser;

[0119] Receive and parse the log to obtain the data change events of the database.

[0120] The event listening parser can determine according to the configuration file that the changes of some tables or fields need to be synchronized across regions, and the changes of other tables are only processed locally in the central region or ignored. The event listening parser supports dynamic configuration adjustment of tables and fields to meet the data synchronization requirements in different business scenarios, and supports the switching between incremental data push and full data push. Figure 3 This is a schematic diagram of the working principle of an event listening parser provided by an embodiment of the present invention. As Figure 3 shown, it includes a database cluster master node and an event listening parser. Here, the event listening parser belongs to the slave node of the database cluster relative to the database cluster master node. The working process of the event listening parser includes:

[0121] 1. Disguise as a slave node: The event listening parser disguises itself as a slave node of the database (MySQL, a relational database management system) cluster, starts the input / output (I / O) thread, and initiates a dump request to the MySQL master node. The master node starts the binlog dump thread and pushes the Binlog log to the I / O thread of this device. In this way, based on the master-slave protocol of MySQL, the event listening parser can monitor the changes of the Binlog log of the master node in the database cluster in real time;

[0122] 2. Event parsing: For insert, update, and delete events, information such as table name, field values, and change types is parsed out.

[0123] 3. Event filtering: It can be configured to determine how to handle events of different tables. Events of tables that are not concerned are directly discarded, improving the processing efficiency.

[0124] 4. If the table is user_entity (related to users), the change information is pushed to the cloud platform deployed in the edge region.

[0125] 5. Data of non-user_entity tables can be locally processed within the cloud platform deployed in the central region (such as only updating the cache of the cloud platform deployed in the central region or executing other logics), and is not pushed to the cloud platform deployed in the edge region.

[0126] In the process of cross-region transmitting the message in the first message queue representing the first data change event to the second message queue in the cloud platform deployed in the edge region, a custom cross-message queue bridge can be written, but this process is relatively complex. Therefore, in the present invention, cross-region transmitting the message in the first message queue representing the first data change event to the second message queue in the cloud platform deployed in the edge region includes:

[0127] Configuring parameters of the message forwarding task in the plugin for message forwarding; wherein, the parameters of the message forwarding task at least include the first message queue, the second message queue, and the connection information between the first message queue and the second message queue.

[0128] Connecting the plugin for message forwarding to the first message queue according to the parameters of the message forwarding task, pulling the message representing the first data change event from the first message queue, and transmitting the message representing the first data change event to the second message queue in the cloud platform deployed in the edge region.

[0129] Meanwhile, in order to achieve message synchronization in multiple edge regions, the second message queue in the parameters of the message forwarding task is located in the cloud platforms deployed in multiple edge regions. Transmitting the message representing the first data change event to the second message queue in the cloud platform deployed in the edge region includes: transmitting the message representing the first data change event to the second message queues in the cloud platforms deployed in multiple edge regions.

[0130] Next, the cross - domain transmission of messages will be further described. In the original single - region architecture, the message queue only needs to transfer messages within the same data center or the same set of message queue clusters. In a multi - region scenario, the cloud platforms deployed in the central region and the cloud platforms deployed in the edge regions each have independent message queue clusters. In order to enable messages to automatically flow from the message queue of the cloud platform deployed in the central region to the message queue of the cloud platform deployed in the edge region, the Shovel mechanism of the message queue is required. The plugin for message forwarding has this Shovel mechanism. After the event listener parser outputs the change event of user_entity, it is first written into the message queue of the cloud platform deployed in the central region. Then, the plugin for message forwarding is used for cross - domain transmission of messages. The specific principle of this plugin for message forwarding is briefly described as follows:

[0131] 1) Shovel mechanism: Configure Shovel through the management terminal of the cloud platform deployed in the central region, with the queue of the cloud platform deployed in the central region as the source and the message queue of the cloud platform deployed in the edge region as the target;

[0132] 2) Shovel continuously monitors the source queue. Once a new message is detected, it will automatically "shovel" it to the target message queue of the cloud platform deployed in the edge region;

[0133] 3) Strategies such as message confirmation, pre - fetch limit, and network retry can be configured to ensure the reliability of messages during cross - network transmission.

[0134] After the cloud platform deployed in the edge region receives the message from the cloud platform deployed in the central region, it is stored in the local queue waiting to be consumed, ensuring the synchronization of messages across message queue clusters.

[0135] Through the above process, the automatic forwarding of messages between cross - message queue clusters is realized, enabling the change of user_entity in the cloud platform deployed in the central region to be "cross - regionally" delivered to the cloud platform deployed in the edge region.

[0136] After the message is cross - domain transmitted to the cloud platform deployed in the edge region through the plugin for message forwarding, in the cloud platform deployed in the edge region, the database does not store user data, but still needs to maintain the same user cache as the cloud platform deployed in the central region. Therefore, it is necessary to refresh the cache of the cloud platform deployed in the edge region. To avoid the risk of cache inconsistency caused by concurrent write operations, the cloud platform deployed in the edge region processes the data in the second cache according to the messages in the second message queue, including: the cloud platform deployed in the edge region parses the messages in the second message queue; and processes the data in the second cache in a single - thread or serial manner according to the parsed messages.

[0137] After the cloud platform deployed in the edge area processes the received messages, there may be messages that are not processed successfully, that is, the cache is not updated successfully, resulting in data inconsistency. To achieve data consistency, after the cloud platform deployed in the edge area processes the data in the second cache according to the messages in the second message queue, it further includes:

[0138] Since the cloud platform deployed in the edge area starts to process the data in the second cache according to the messages in the second message queue, it obtains unprocessed message events within a preset duration and stores the unprocessed message events in the message queue used to represent processing failures in the second message queue; processes the data in the second cache according to the messages in the message queue used to represent processing failures.

[0139] There is no limit to the preset duration, which is determined according to the actual situation. In addition, the cloud platform deployed in the edge area processes the data in the second cache according to the messages in the second message queue, including:

[0140] The cloud platform deployed in the edge area obtains the priority order of the messages in the second message queue and the timestamps of receiving the messages;

[0141] Combines the priority order of the messages in the second message queue and the timestamp order of receiving the messages and processes the data in the second cache according to the content of the messages in the second message queue.

[0142] The cloud platform deployed in the edge area processes based on the priority or timestamp of the messages, ensuring the real-time and accuracy of cache updates.

[0143] The above process can be called the cache refresh of the cloud platform deployed in the edge area. In implementation, a message consumer (such as a microservice, iregion, iresource-common) is deployed in the cloud platform deployed in the edge area to consume the user_entity update messages from the message queue of the cloud platform deployed in the edge area. The specific process of cache refresh is as follows:

[0144] 1. Consume messages: When receiving a message, parse information such as the user primary key identifier (Identity, ID) and change type (insert, update, delete) in the message;

[0145] 2. Single-threaded cache refresh: To avoid concurrency problems, a single thread is used to perform operations on the cache:

[0146] 1) If it is a deletion event, delete the corresponding cache key;

[0147] 2) If it is an update / insert event, update the data in the cache;

[0148] 3. Failure Retry: If the write cache fails, write the event to the failure queue of the local message queue (or other custom retry queues), and perform retries or automatic error correction later to ensure eventual consistency.

[0149] Since the cloud platform deployed in the edge region does not have the user_entity data table and cannot directly drive cache updates based on local Binlog, the cross-region message queue event has become the only reliable source of incremental updates. The cache of the cross-region cloud platform is updated according to the message queue.

[0150] In practice, there may be data changes in the database of the cloud platform deployed in the edge region. To ensure the data consistency between the database of the cloud platform deployed in the edge region and the cache of the cloud platform deployed in the edge region, the multi-region cloud platform data consistency processing method further includes:

[0151] The cloud platform deployed in the edge region receives the second data change event of the second database; wherein, the second database is located in the cloud platform deployed in the edge region; process the data in the second cache according to the second data change event.

[0152] The cloud platform deployed in the edge region monitors the changes of the second database in the same way as described above for monitoring the changes of the first database in the cloud platform deployed in the edge region, both through the event listening parser for monitoring. In addition, to increase the probability of successfully capturing the database change event, there are multiple event listening parsers in the cloud platform deployed in the edge region for monitoring the second database. The principle of how the event listening parser implements monitoring has been described above and will not be elaborated here.

[0153] In practice, there may be situations such as network instability and offline in the cloud platform deployed in the edge region, resulting in some cache data updates being missed. In this case, to ensure data consistency, the multi-region cloud platform data consistency processing method further includes:

[0154] The cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database;

[0155] Empty the data in the second cache, and store the data to be flushed into the second cache into the second cache;

[0156] Obtain the target information from the cloud platform deployed in the central region and store the target information into the second cache; wherein, the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region.

[0157] The above process can be called the full refresh (also known as cache reset) of the cache of the cloud platform deployed in the edge region. Figure 4 It is a flowchart of a data transmission method provided by an embodiment of the present invention, asFigure 4 As shown, the method includes:

[0158] S13: The first database enables log listening;

[0159] S14: The event listening parser parses the event and pushes the event to the message queue of the cloud platform deployed in the central region;

[0160] S15: The message queue of the cloud platform deployed in the central region shovels the message to the message queue of the cloud platform deployed in the edge region through the shovel mechanism;

[0161] S16: After message consumption, refresh the cache of the cloud platform deployed in the edge region;

[0162] S17: Periodically / start a full refresh to actively obtain data from the message queue of the cloud platform deployed in the central region.

[0163] If a full refresh is performed in all scenarios, a large amount of resources will be consumed. Therefore, when it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is a preset moment, or when it is detected that the user issues an instruction for triggering a cache reset, clear the data in the second cache and store the data to be flushed into the second cache in the second cache.

[0164] The process of the full refresh of the cloud platform deployed in the edge region mentioned above is described again below:

[0165] 1. Trigger scenarios:

[0166] 1) When the system starts;

[0167] 2) Scheduled tasks (such as every day at midnight);

[0168] 3) Manual trigger (when there are large-scale inconsistencies or maintenance needs);

[0169] 2. Execution process:

[0170] 1) Call the application programming interface (API) of the central Region to batch obtain the complete user_entity data;

[0171] 2) Clear the cache keys related to user_entity in the second cache;

[0172] 3) Write the full amount of data into the second cache to ensure consistency with the cloud platform deployed in the central region.

[0173] If new incremental changes occur during the full refresh, secondary compensation or replay can be performed after the refresh to ensure the data is up-to-date.

[0174] Through full reset, the incremental updates that may be missed by the cloud platform deployed in the central region during offline or network anomalies can be corrected, so that the cache of the cloud platform deployed in the central region is finally consistent with the database of the cloud platform deployed in the central region.

[0175] The above describes the full refresh of the cache of the cloud platform deployed in the edge region. In practice, the cloud platform deployed in the central region may also experience network instability, offline, etc., resulting in some cache data updates being missed. In this case, to ensure data consistency, the multi-region cloud platform data consistency processing method also includes:

[0176] Obtain the data to be flushed into the first cache from the first database;

[0177] Empty the data in the first cache and store the data to be flushed into the first cache in the first cache.

[0178] Similarly, to reduce the consumed resources, when it is detected that the multi-region cloud platform is started, or when it is detected that the current time is the preset time, or when it is detected that the user issues an instruction for triggering the cache reset, empty the data in the first cache and store the data to be flushed into the first cache in the first cache.

[0179] The above process realizes the full refresh of the second cache of the cloud platform deployed in the edge region and the full refresh of the first cache of the cloud platform deployed in the central region. If the corresponding cache is updated according to the change event of the database (referred to as real-time update cache) during the full refresh process, it may affect the integrity of the data. Therefore, after obtaining the data to be flushed into the first cache from the first database, before emptying the data in the first cache and storing the data to be flushed into the first cache in the first cache, it also includes:

[0180] Stop processing the data in the first cache according to the first data change event.

[0181] After emptying the data in the first cache and storing the data to be flushed into the first cache in the first cache, it also includes:

[0182] Process the data in the first cache according to the first data change event.

[0183] That is, when performing a full refresh of the cache of the cloud platform deployed in the central region, block the task of real-time updating the cache, and only perform the full refresh (only execute the process of the timed task execution). After the full refresh is completed, continue to perform the real-time update cache operation, ensuring the integrity of the cache data.

[0184] Similarly, in order to ensure the integrity of the cached data of the cloud platform deployed in the central region, after the cloud platform deployed in the edge region obtains the data to be written into the second cache from the second database, before clearing the data in the second cache and storing the data to be written into the second cache into the second cache, it further includes:

[0185] Stop processing the data in the second cache according to the second data change event.

[0186] After obtaining the target information from the cloud platform deployed in the central region and storing the target information into the second cache, it further includes:

[0187] Process the data in the second cache according to the second data change event.

[0188] That is, when performing a full refresh of the cache of the cloud platform deployed in the edge region, block the task of real-time cache update, and only perform a full refresh (only execute the process of the timed task). After the full refresh is completed, continue with the real-time cache update operation, ensuring the integrity of the cached data.

[0189] In practice, there may be data change events that are not processed successfully. In this case, in order to ensure data consistency, after processing the data in the first cache according to the first data change event, it further includes:

[0190] Obtain the first data change event that was not processed successfully;

[0191] Store the first data change event that was not processed successfully into the message queue for representing processing failure in the first message queue;

[0192] Start an independent thread to process the first data change event that was not processed successfully in the message queue for representing processing failure;

[0193] Delete the cached data corresponding to the first data change event that was not processed successfully from the first cache.

[0194] After processing the data in the second cache according to the second data change event, it further includes:

[0195] Obtain the second data change event that was not processed successfully;

[0196] Store the second data change event that was not processed successfully into the message queue for representing processing failure in the second message queue;

[0197] Start an independent thread to process the second data change event that was not processed successfully in the message queue for representing processing failure;

[0198] Delete the cached data corresponding to the second data change event that was not processed successfully from the second cache.

[0199] In this method, for data change events that are not successfully processed, they are stored in a message queue used to represent processing failures, an independent thread is started for processing, and the corresponding data is deleted from the cache to ensure the consistency of the final data.

[0200] When a user queries data from a cloud platform deployed in a peripheral region or from a cloud platform deployed in a central region, in order to improve query performance and simplify system design, the multi-region cloud platform data consistency processing method further includes:

[0201] The cloud platform deployed in the peripheral region receives a request sent by the user through a unified interface (such as a Software Development Kit (SDK)) to represent querying data from the second cache; if it is detected that the data corresponding to the request exists in the second cache, the data corresponding to the request is returned to the user; if it is detected that the data corresponding to the request does not exist in the second cache, the application programming interface of the cloud platform deployed in the central region is called to obtain the data corresponding to the request;

[0202] Or, receive a request sent by the user through a unified interface to represent querying data from the first cache;

[0203] Return the data corresponding to the request that exists in the first cache to the user.

[0204] The above process is called the cache query process. For the query of user data by the application service, the cloud platform deployed in the peripheral region provides a unified SDK for cache query. The caller only needs to provide keyword fields such as the user ID, and the SDK will perform the following steps:

[0205] 1. First, access the cache of the cloud platform deployed in the peripheral region;

[0206] 2. If the cache is hit, directly return the data;

[0207] 3. If not hit, call the interface of the cloud platform deployed in the central region to further confirm whether the data exists and update the cache.

[0208] In a multi-region scenario, if the cloud platform deployed in the peripheral region does not have user information in the local database, the query interface of the cloud platform deployed in the central region can be directly called according to business requirements to complete the "penetration" query. In this way, the burden on developers can be further reduced and the system consistency can be improved.

[0209] The above text describes a multi-region cloud platform data consistency processing method applied to a cloud platform deployed in a central region. This embodiment also provides a multi-region cloud platform data consistency processing method applied to a cloud platform deployed in a peripheral region. The method includes:

[0210] Receive the messages in the second message queue; wherein, the messages in the second message queue are the first data change events of the first database obtained by the cloud platform deployed in the central region; process the data in the first cache according to the first data change events, and store the first data change events into the first message queue; the messages used to represent the first data change events in the first message queue are obtained by cross-region transmission to the second message queue in the cloud platform deployed in the edge region; wherein, the first database, the first cache and the first message queue are all located in the cloud platform deployed in the central region;

[0211] Process the data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0212] The method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the edge region provided in this embodiment has the same or corresponding technical features as the method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the central region described above. The embodiments of the method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the central region have been described in detail above, and the embodiments of the method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the edge region will not be elaborated here, and the effects are the same.

[0213] In order to enable those skilled in the art to better understand the overall solution of the data consistency method of the multi-region deployed cloud platform of the present invention, the following will continue to be described in conjunction with the accompanying drawings and specific embodiments. Figure 5 It is a schematic diagram of the overall data consistency method of a multi-region deployed cloud platform provided by an embodiment of the present invention.

[0214] In the cloud platform deployed in the central region:

[0215] 1.1. Monitor the log events of the first database and discard the events that do not need to be flushed into the cache;

[0216] 1.2. The master node in the multi-copy scenario updates the cache in real time;

[0217] The above steps 1.1 to 1.2 are called the steps of updating the cache in real time.

[0218] 1.3. The events of cache writing failure are temporarily stored in the message queue;

[0219] 1.4. Start an independent thread to consume the events and delete the cache data corresponding to the writing failure events;

[0220] 1.5. Monitor the user-related log events of the first database.

[0221] Push the monitored user-related log events as messages to the first message queue through the microservice (iregion); transfer them cross-region to the second message queue in the cloud platform deployed in the edge region through the shovel mechanism, and input them into the second cache through microservice one (iregion) and microservice two (iresource-common).

[0222] 2.1. Read the data that needs to be flushed into the cache from the first database;

[0223] 2.2. Clear the cache and flush the read data into the cache.

[0224] Steps 2.1 to 2.2 can be called the full refresh task of the cache in the cloud platform deployed in the central region. It should be noted that during the execution of the full refresh task, the real-time update of the cache is blocked.

[0225] In the cloud platform deployed in the edge region:

[0226] 1.1. Monitor the log events of the second database and discard the events that do not need to be flushed into the cache;

[0227] 1.2. The master node in the multi-copy scenario updates the cache in real time.

[0228] The above steps 1.1 to 1.2 are called the steps for real-time cache update.

[0229] 1.3. Events with cache write failures are temporarily stored in the message queue;

[0230] 1.4. Start an independent thread to consume the events and delete the cache data corresponding to the write failure events. The above steps 1.1 to 1.4 are called the steps for real-time cache update;

[0231] 2.1. Read the data that needs to be flushed into the cache from the second database;

[0232] 2.2. Clear the cache and flush the read data into the cache;

[0233] 2.3. Clear the user cache, query the user information in the central region, and flush it into the cache.

[0234] Steps 2.1 to 2.3 can be called the full refresh task of the cache in the cloud platform deployed in the edge region. It should be noted that during the execution of the full refresh task, the real-time update of the cache is blocked.

[0235] The overall idea is as follows:

[0236] 1. The database (i.e., the first database) in the cloud platform deployed in the central region enables the Binlog function to record all data changes.

[0237] 2. Event listening parser: A listening program is deployed on the cloud platform deployed in the central region to parse Binlog events in real time and obtain information such as table names, primary keys, and change types.

[0238] 1) Different processing strategies can be adopted for different tables;

[0239] 2) If it is a user_entity table event related to cross-region synchronization, a message is pushed to the edge Region;

[0240] 3) Data of other tables is only processed locally on the cloud platform deployed in the central region (not pushed).

[0241] 3. Cross-region message distribution: The captured user_entity table events are written into the first message queue of the cloud platform deployed in the central region, and then the shoveling mechanism of the first message queue is used to "shovel" the messages to the second message queues of the cloud platforms deployed in each edge region.

[0242] 4. Cache refresh on the cloud platform deployed in the edge region: The cloud platform deployed in the edge region listens to the local second message queue. After receiving the incremental update message from the cloud platform deployed in the central region, it updates / deletes the corresponding data in the second cache.

[0243] 5. Full cache refresh in the edge Region: The cloud platform deployed in the edge region actively pulls the complete user_entity data from the cloud platform deployed in the central region at regular intervals or when starting up. After clearing the user-related cache in the second cache, it reloads it to correct possible omissions or deviations.

[0244] 6. Cache queryer: When the application queries user information on the cloud platform deployed in the edge region, it can first read the cache; if the cache is not hit, it calls the user information API of the cloud platform deployed in the central region.

[0245] Through this method, the goal of keeping the user data on the cloud platform deployed in the edge region consistent with that on the cloud platform deployed in the central region without database tables is achieved. Specifically:

[0246] 1. The cloud platform deployed in the central region listens to Binlog:

[0247] Parse the real-time add, delete, and modify logs of the database;

[0248] When the table name is detected as user_entity, wrap the message body (primary key, change type, field data) and write it into the user_entity_update queue in the first message queue of the cloud platform deployed in the central region.

[0249] 2. Configuration of the shoveling mechanism of the first message queue:

[0250] Set up Shovel in the cloud platform deployed in the central region. The source end is the user_entity_update queue, and the target end is the user_entity_edge queue in the edge Region.

[0251] When Shovel detects a new message, it automatically grabs and forwards it to the message queue of the cloud platform deployed in the edge region without manual intervention.

[0252] 3. The cloud platform deployed in the edge region receives and refreshes the cache:

[0253] Local consumers subscribe to the user_entity_edge queue to receive incremental update messages;

[0254] According to the event type, perform insert, update, or delete operations on the second cache;

[0255] If the operation fails, record it in the retry queue or log and recover later.

[0256] 4. Full refresh of the cloud platform deployed in the edge region:

[0257] The daily early morning scheduled task is triggered, or the following steps are executed when the system starts:

[0258] 1) Call the API interface of the cloud platform deployed in the central region to obtain all user information;

[0259] 2) Clear the cache of user_entity in the second cache;

[0260] 3) Write the data into the second cache one by one;

[0261] 4) If incremental events arrive during the refresh, they can be replayed again after the refresh is completed to avoid overwriting updates.

[0262] Through the above process, the cloud platform deployed in the edge region can timely obtain the changes in user data and maintain the unity of the cache; even if message loss or the edge region goes offline occurs, it can quickly recover to the correct state through a full reset. A set of user systems is shared across multiple regions. The edge region does not need to maintain a complete set of user data and can obtain user data consistent with the central region only through the cache and message queue, reducing the deployment and maintenance costs and achieving better unified user management at the same time; Real-time incremental + regular full volume: Real-time updates are achieved through Binlog monitoring + cross-region message pushing. At the same time, full refresh is performed regularly or at startup to prevent omissions, greatly improving the system availability and fault tolerance; The shovel mechanism of the message queue synchronizes across clusters: There is no need to write a custom cross-message queue bridge. The shovel mechanism provides secure and stable message "shoveling" and can be smoothly extended to more regions or multi-active data center scenarios; Reduce the complexity of application development: The edge region or other callers only need to access the cache query (or unified SDK), avoiding frequent cross-region access to the central database, improving the query performance and simplifying the system design.

[0263] In the above embodiments, the method for data consistency of the cloud platform with multi-region deployment is described in detail. The present invention also provides embodiments corresponding to the data consistency device of the cloud platform with multi-region deployment.

[0264] An embodiment of the present invention provides a device for processing data consistency of a multi-region cloud platform. The device is applied to the cloud platform deployed in the central region and includes:

[0265] An acquisition module, configured to acquire a first data change event of a first database;

[0266] A processing and storage module, configured to process the data in the first cache according to the first data change event and store the first data change event in a first message queue, where the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region;

[0267] A transmission module, configured to transmit the message representing the first data change event in the first message queue across domains to a second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, where the second cache is located in the cloud platform deployed in the edge region.

[0268] In some embodiments, the device for processing data consistency of the multi-region cloud platform further includes:

[0269] A detection and trigger module, configured to trigger the storage module in the processing and storage module when it is detected that the first data change event is a preset event, where the storage module is configured to store the first data change event in the first message queue;

[0270] A processing module, configured to locally process a first data change event when it is detected that the first data change event is not a preset event.

[0271] In some embodiments, the multi-region cloud platform data consistency processing device further includes a first processing module, which is configured to process the data in the second cache according to the messages in the second message queue by the cloud platform deployed in the edge region.

[0272] The first processing module includes:

[0273] A parsing and processing module, configured to parse the messages in the second message queue; process the data in the second cache in a single-threaded or serial manner according to the parsed messages;

[0274] The multi-region cloud platform data consistency processing device further includes:

[0275] A first storage module, configured to, since the cloud platform deployed in the edge region starts to process the data in the second cache according to the messages in the second message queue, obtain unprocessed message events within a preset duration and store the unprocessed message events into the message queue for representing processing failures in the second message queue; process the data in the second cache according to the messages in the message queue for representing processing failures.

[0276] The first processing module includes:

[0277] A first obtaining module, configured to obtain the priority order of the messages in the second message queue and the timestamps of receiving the messages by the cloud platform deployed in the edge region;

[0278] A second processing module, configured to process the data in the second cache by combining the priority order of the messages in the second message queue and the timestamp order of receiving the messages and according to the content of the messages in the second message queue.

[0279] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0280] A first receiving module, configured to receive a second data change event of a second database by the cloud platform deployed in the edge region; wherein, the second database is located in the cloud platform deployed in the edge region;

[0281] A third processing module, configured to process the data in the second cache according to the second data change event.

[0282] In some embodiments, the multi-region cloud platform data consistency processing device includes a second obtaining module, configured to obtain a data change event of a database.

[0283] The second obtaining module is specifically configured to obtain a data change event of a database through a plurality of event listening parsers.

[0284] In some embodiments, the multi-region cloud platform data consistency processing device includes a third acquisition module for an event listening parser to acquire data change events of a database. The third acquisition module specifically includes:

[0285] An enabling module for the event listening parser to act as a slave node of the database cluster and enable input / output threads;

[0286] A request module for initiating a data export request to the master node in the database cluster so that the master node in the database cluster enables a log export thread and uses the log to respond to the data export request of the event listening parser;

[0287] A second receiving module for receiving and parsing the log to obtain data change events of the database.

[0288] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0289] A third acquisition module for the cloud platform deployed in the edge region to acquire data to be flushed into the second cache from a second database;

[0290] A first clearing and storing module for clearing the data in the second cache and storing the data to be flushed into the second cache in the second cache;

[0291] An acquisition and storage module for acquiring target information from the cloud platform deployed in the central region and storing the target information in the second cache; wherein, the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region.

[0292] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0293] A fourth acquisition module for acquiring data to be flushed into the first cache from a first database;

[0294] A first clearing and storing module for clearing the data in the first cache and storing the data to be flushed into the first cache in the first cache.

[0295] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0296] A first stopping module for stopping processing the data in the first cache according to a first data change event.

[0297] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0298] A second stopping module for stopping processing the data in the second cache according to a second data change event.

[0299] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0300] A fifth acquisition module, configured to acquire first data change events that have not been successfully processed;

[0301] A second storage module, configured to store the first data change events that have not been successfully processed into a message queue for representing processing failures in the first message queue;

[0302] A fourth processing module, configured to start an independent thread to process the first data change events that have not been successfully processed in the message queue for representing processing failures;

[0303] A first deletion module, configured to delete the cached data corresponding to the first data change events that have not been successfully processed from the first cache.

[0304] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0305] A sixth acquisition module, configured to acquire second data change events that have not been successfully processed;

[0306] A third storage module, configured to store the second data change events that have not been successfully processed into a message queue for representing processing failures in the second message queue;

[0307] A fifth processing module, configured to start an independent thread to process the second data change events that have not been successfully processed in the message queue for representing processing failures;

[0308] A second deletion module, configured to delete the cached data corresponding to the second data change events that have not been successfully processed from the second cache.

[0309] In some embodiments, the multi-region cloud platform data consistency processing device further includes:

[0310] A first request processing module, configured to receive, through a unified interface, a request sent by a user for querying data from the second cache by a cloud platform deployed in an edge region; if it is detected that the data corresponding to the request exists in the second cache, return the data corresponding to the request to the user; if it is detected that the data corresponding to the request does not exist in the second cache, call the application programming interface of the cloud platform deployed in the central region to acquire the data corresponding to the request;

[0311] Or, a second request processing module, configured to receive, through a unified interface, a request sent by a user for querying data from the first cache; return the data corresponding to the request that exists in the first cache to the user.

[0312] The embodiment of the present invention further provides a multi-region cloud platform data consistency processing device applied to a cloud platform deployed in an edge region. The device includes:

[0313] A third receiving module, configured to receive messages in the second message queue; wherein, the messages in the second message queue are obtained by the cloud platform deployed in the central region from the first data change event of the first database; process the data in the first cache according to the first data change event, and store the first data change event into the first message queue; and cross-region transmit the messages in the first message queue representing the first data change event to the second message queue in the cloud platform deployed in the edge region; wherein, the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region;

[0314] A sixth processing module, configured to process the data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0315] Since the embodiments of the device part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the device part, which will not be elaborated here.

[0316] Figure 6 This is the structural diagram of the multi-region cloud platform system provided by the embodiment of the present invention. From a hardware perspective, as Figure 6 shown, the multi-region cloud platform system includes:

[0317] A memory 20, configured to store a computer program;

[0318] A processor 21, configured to implement the steps of the multi-region cloud platform data consistency processing method as mentioned in the above embodiments when executing the computer program.

[0319] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array. The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may further include an artificial intelligence (AI) processor, and the AI processor is used to process computational operations related to machine learning.

[0320] The memory 20 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 20 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201. After the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the multi-regional cloud platform data consistency processing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may further include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the multi-regional cloud platform data consistency processing method mentioned above.

[0321] In some embodiments, the multi-regional cloud platform system may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0322] Those skilled in the art can understand that Figure 6 the structure shown in

[0323] The multi-region cloud platform system provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, the following method can be implemented: a method for processing data consistency of the multi-region cloud platform, with the same effect.

[0324] The embodiment of the present invention also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by the processor, the steps of the above-mentioned method for processing data consistency of the multi-region cloud platform are implemented.

[0325] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps recorded in the above-mentioned method embodiment (which can be the method corresponding to the cloud platform deployed in the central region, or the method corresponding to the cloud platform deployed in the edge region, or the method corresponding to both the cloud platform side deployed in the central region and the cloud platform side deployed in the edge region) are implemented.

[0326] It can be understood that if the method in the above-mentioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0327] The computer-readable storage medium provided by the present invention includes the above-mentioned method for processing data consistency of the multi-region cloud platform, with the same effect.

[0328] The above has introduced in detail the method, device, product, and system for processing data consistency of the multi-region cloud platform provided by the present invention. The various embodiments in the specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0329] It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

Claims

1. A method for processing data consistency in a multi-region cloud platform, characterized in that A cloud platform applied to central region deployment; the method includes: Obtain the first data change event of the first database; Process the data in the first cache according to the first data change event, and store the first data change event into the first message queue; wherein, the first database, the first cache and the first message queue are all located in the cloud platform deployed in the central region; Transmit the message used to represent the first data change event in the first message queue across domains to the second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region; The transmitting the message used to represent the first data change event in the first message queue across domains to the second message queue in the cloud platform deployed in the edge region includes: Configure the parameters of the message forwarding task in the plugin for message forwarding; wherein, the parameters of the message forwarding task at least include the first message queue, the second message queue, and the connection information between the first message queue and the second message queue; Connect the plugin for message forwarding to the first message queue according to the parameters of the message forwarding task, pull the message used to represent the first data change event from the first message queue, and transmit the message used to represent the first data change event to the second message queue in the cloud platform deployed in the edge region.

2. The method for processing data consistency of a multi-region cloud platform according to claim 1, wherein Before storing the first data change event into the first message queue, it further includes: In the case where it is detected that the first data change event is a preset event, enter the step of storing the first data change event into the first message queue; In the case where it is detected that the first data change event is not a preset event, process the first data change event locally.

3. The method for processing data consistency of a multi-region cloud platform according to claim 1, wherein The second message queue in the parameters of the message forwarding task is located in multiple cloud platforms deployed in the edge region; The transmitting the message used to represent the first data change event to the second message queue in the cloud platform deployed in the edge region includes: Transmit the message used to represent the first data change event to the second message queue in multiple cloud platforms deployed in the edge region.

4. The method for processing data consistency of a multi-region cloud platform according to claim 1, characterized in that, The cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, including: The cloud platform deployed in the edge region parses the message in the second message queue; processes the data in the second cache in a single-threaded or serial manner and according to the parsed message; After the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, it further includes: The cloud platform deployed in the edge region, starting from processing the data in the second cache according to the message in the second message queue, obtains the unprocessed message events within a preset duration, and stores the unprocessed message events into the message queue used to represent processing failures in the second message queue; processes the data in the second cache according to the message in the message queue used to represent processing failures.

5. The method for processing data consistency of a multi-region cloud platform according to claim 1, wherein The cloud platform deployed in the edge region processes the data in the second cache according to the messages in the second message queue, including: The cloud platform deployed in the edge region obtains the priority order of the messages in the second message queue and the timestamps of receiving the messages; Combining the priority order of the messages in the second message queue and the timestamp order of receiving the messages, and processing the data in the second cache according to the content of the messages in the second message queue.

6. The method for processing data consistency of a multi-regional cloud platform according to claim 1, wherein The method further includes: The cloud platform deployed in the edge region receives the second data change event of the second database; wherein, the second database is located in the cloud platform deployed in the edge region; Processing the data in the second cache according to the second data change event.

7. The method for processing data consistency of a multi-region cloud platform according to claim 6, wherein There are multiple event listening and parsing resolvers in the cloud platform deployed in the central region for monitoring the first database, and there are multiple event listening and parsing resolvers in the cloud platform deployed in the edge region for monitoring the second database; Obtaining the data change event of the database includes: Obtaining the data change event of the database through multiple event listening and parsing resolvers; Processing the data in the cache according to the data change event includes: Obtaining the main event listening and parsing resolver from multiple event listening and parsing resolvers; Processing the data in the cache according to the data change event obtained by the main event listening and parsing resolver.

8. The method for processing data consistency of a multi-region cloud platform according to claim 7, wherein The event listening and parsing resolver obtaining the data change event of the database includes: The event listening and parsing resolver acts as a slave node of the database cluster and starts the input / output thread; Sending a data export request to the master node in the database cluster so that the master node in the database cluster starts the log export thread and uses the log to respond to the data export request of the event listening and parsing resolver; Receiving and parsing the log to obtain the data change event of the database.

9. The method for processing data consistency of a multi-region cloud platform according to claim 6, wherein The method further includes: The cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database; Clearing the data in the second cache and storing the data to be flushed into the second cache in the second cache; Obtaining the target information from the cloud platform deployed in the central region and storing the target information in the second cache; wherein, the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region.

10. The method for processing data consistency of a multi-region cloud platform according to claim 9, characterized in that, The method further includes: Obtaining the data to be flushed into the first cache from the first database; Clearing the data in the first cache and storing the data to be flushed into the first cache in the first cache.

11. The method for processing data consistency of a multi-region cloud platform according to claim 10, wherein The clearing the data in the second cache and storing the data to be flushed into the second cache in the second cache, or the clearing the data in the first cache and storing the data to be flushed into the first cache in the first cache includes: When it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is a preset moment, or when it is detected that the user issues an instruction for triggering the reset of the cache, clearing the data in the second cache and storing the data to be flushed into the second cache in the second cache, or clearing the data in the first cache and storing the data to be flushed into the first cache in the first cache.

12. The method for processing data consistency of a multi-region cloud platform according to claim 10, wherein After obtaining the data to be flushed into the first cache from the first database, before clearing the data in the first cache and storing the data to be flushed into the first cache into the first cache, it further includes: Stop processing the data in the first cache according to the first data change event; After clearing the data in the first cache and storing the data to be flushed into the first cache into the first cache, it further includes: Process the data in the first cache according to the first data change event; After the cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database, before clearing the data in the second cache and storing the data to be flushed into the second cache into the second cache, it further includes: Stop processing the data in the second cache according to the second data change event; After obtaining the target information from the cloud platform deployed in the central region and storing the target information into the second cache, it further includes: Process the data in the second cache according to the second data change event.

13. The method for processing data consistency of a multi-region cloud platform according to claim 6, characterized in that After processing the data in the first cache according to the first data change event, it further includes: Obtain the first data change events that have not been successfully processed; Store the first data change events that have not been successfully processed into the message queue for representing processing failures in the first message queue; Start an independent thread to process the first data change events that have not been successfully processed in the message queue for representing processing failures; Delete the cache data corresponding to the first data change events that have not been successfully processed from the first cache; After processing the data in the second cache according to the second data change event, it further includes: Obtain the second data change events that have not been successfully processed; Store the second data change events that have not been successfully processed into the message queue for representing processing failures in the second message queue; Start an independent thread to process the second data change events that have not been successfully processed in the message queue for representing processing failures; Delete the cache data corresponding to the second data change events that have not been successfully processed from the second cache.

14. The method for processing data consistency of a multi-regional cloud platform according to any one of claims 1 to 13, characterized in that, The method further includes: The cloud platform deployed in the edge region receives a request from the user for querying data from the second cache through a unified interface; if it is detected that the data corresponding to the request exists in the second cache, the data corresponding to the request is returned to the user; if it is detected that the data corresponding to the request does not exist in the second cache, the application programming interface of the cloud platform deployed in the central region is called to obtain the data corresponding to the request; Or, receive a request from the user for querying data from the first cache through a unified interface; Return the data corresponding to the request existing in the first cache to the user.

15. A method for processing data consistency in a multi-region cloud platform, characterized in that, Applied to the cloud platform deployed in the edge region, the method includes: Receive messages in the second message queue; wherein, the messages in the second message queue are obtained by the cloud platform deployed in the central region from the first data change events of the first database; process the data in the first cache according to the first data change events, and store the first data change events into the first message queue; the messages in the first message queue used to represent the first data change events are obtained by cross-region transmission to the second message queue in the cloud platform deployed in the edge region; wherein, the first database, the first cache and the first message queue are all located in the cloud platform deployed in the central region; the cross-region transmission of the messages in the first message queue used to represent the first data change events to the second message queue in the cloud platform deployed in the edge region includes: configuring the parameters of the message forwarding task in the plugin for message forwarding; the parameters of the message forwarding task at least include the connection information between the first message queue, the second message queue, the first message queue and the second message queue; connect the plugin for message forwarding to the first message queue according to the parameters of the message forwarding task, pull the messages used to represent the first data change events from the first message queue, and transmit the messages used to represent the first data change events to the second message queue in the cloud platform deployed in the edge region; Process the data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

16. A multi-region cloud platform data consistency processing device, characterized in that, Applied to the cloud platform deployed in the central region; including: An acquisition module, configured to acquire the first data change events of the first database; A processing and storage module, configured to process the data in the first cache according to the first data change events, and store the first data change events into the first message queue; wherein, the first database, the first cache and the first message queue are all located in the cloud platform deployed in the central region; A transmission module, configured to cross-region transmit the messages in the first message queue used to represent the first data change events to the second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region can process the data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region; The transmission module is specifically configured to: Configure the parameters of the message forwarding task in the plugin for message forwarding; wherein, the parameters of the message forwarding task at least include the connection information between the first message queue, the second message queue, the first message queue and the second message queue; Connect the plugin for message forwarding to the first message queue according to the parameters of the message forwarding task, pull the messages used to represent the first data change events from the first message queue, and transmit the messages used to represent the first data change events to the second message queue in the cloud platform deployed in the edge region.

17. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by a processor, the steps of the multi-region cloud platform data consistency processing method according to any one of claims 1 to 15 are implemented.

18. A multi-region cloud platform system, characterized in that, Including: A memory for storing a computer program; A processor for implementing the steps of the multi-region cloud platform data consistency processing method according to any one of claims 1 to 15 when executing the computer program.

19. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the multi-region cloud platform data consistency processing method according to any one of claims 1 to 15 are implemented.

Images