Multi-region cloud platform data consistency processing method, device, product and system

Abstract

The invention discloses a multi-region cloud platform data consistency processing method, device, product and system, and relates to the field of cloud computing. In the method, after data change of a central database is detected, the data is converted into messages in a message queue, the messages are transmitted to a message queue of an edge region cloud platform in a cross-domain manner, and the edge region cloud platform updates own cache according to the messages in the message queue. The data consistency of the database of the cloud platform deployed in the central region and the cached data of the cloud platform deployed in the edge region is ensured; and after the data change of the database of the cloud platform deployed in the central region is detected, the data in the cache of the cloud platform deployed in the central region is updated, so that the data consistency of the cache of the cloud platform deployed in the central region and the database of the cloud platform deployed in the central region is ensured. According to the method, the consistency of the data of the database and the cached data of the cloud platform deployed in the central region and the cached data of the cloud platform deployed in the edge region is realized.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to a method, device, product and system for processing data consistency in a multi-region cloud platform. Background Art

[0002] With the continuous evolution of cloud platform services, there is a need for multi-region (Region) deployment. All data, including user information, resource information, etc., is maintained in the central region (Center Region), while only necessary functions are deployed in each edge region (Edge Region) to provide services. Data inconsistency in a multi-region cloud platform may lead to business errors, and in severe cases, may even cause system failures.

[0003] Related techniques for ensuring data consistency include the dual-write mode technique and the asynchronous cache update technique. In the dual-write mode technique, the database and the cache are written simultaneously. However, in a multi-region scenario, due to security considerations, the cache services in each region are not exposed externally. Therefore, the central cloud platform cannot directly write data changes to the cache of the edge cloud platform. In the asynchronous cache update technique, in a multi-region scenario, since the central database cannot connect to the message queue service in the edge region, the central database cannot directly notify the edge cloud platform. It can be seen that the related techniques for ensuring data consistency still cannot guarantee data consistency in a multi-region cloud platform, and the risks of business errors and system failures caused by data inconsistency still exist.

[0004] Therefore, it can be seen that how to ensure data consistency in a multi-region cloud platform is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device, product and system for processing data consistency in a multi-region cloud platform to solve the technical problem that the related techniques for ensuring data consistency cannot guarantee data consistency in a multi-region cloud platform.

[0006] To solve the above technical problem, the present invention provides a method for processing data consistency in a multi-region cloud platform, which is applied to a cloud platform deployed in a central region; the method includes: Obtain a first data change event of a first database; Process data in a first cache according to the first data change event, and store the first data change event in a first message queue; wherein, the first database, the first cache and the first message queue are all located in the cloud platform deployed in the central region; Transmit the message in the first message queue that characterizes the first data change event across domains to the second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0007] On the one hand, before storing the first data change event into the first message queue, it further includes: In the case where it is detected that the first data change event is a preset event, enter the step of storing the first data change event into the first message queue; In the case where it is detected that the first data change event is not a preset event, process the first data change event locally.

[0008] On the other hand, the transmitting the message in the first message queue that characterizes the first data change event across domains to the second message queue in the cloud platform deployed in the edge region includes: Configure the parameters of the message forwarding task in the plugin for message forwarding; wherein, the parameters of the message forwarding task at least include the first message queue, the second message queue, and the connection information between the first message queue and the second message queue; Connect the plugin for message forwarding to the first message queue according to the parameters of the message forwarding task, pull the message that characterizes the first data change event from the first message queue, and transmit the message that characterizes the first data change event to the second message queue in the cloud platform deployed in the edge region.

[0009] On the other hand, the second message queue in the parameters of the message forwarding task is located in multiple cloud platforms deployed in the edge region; The transmitting the message that characterizes the first data change event to the second message queue in the cloud platform deployed in the edge region includes: Transmit the message that characterizes the first data change event to the second message queue in multiple cloud platforms deployed in the edge region.

[0010] On the other hand, the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue includes: The cloud platform deployed in the edge region parses the message in the second message queue; processes the data in the second cache in a single-threaded or serial manner and according to the parsed message; After the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, it further includes: The cloud platform deployed in the edge region starts to process the data in the second cache according to the messages in the second message queue, obtains unprocessed message events within a preset duration, and stores the unprocessed message events into the message queue for representing processing failures in the second message queue; processes the data in the second cache according to the messages in the message queue for representing processing failures.

[0011] On the other hand, the cloud platform deployed in the edge region processes the data in the second cache according to the messages in the second message queue, including: The cloud platform deployed in the edge region obtains the priority order of the messages in the second message queue and the timestamps of receiving the messages; Combines the priority order of the messages in the second message queue and the timestamp order of receiving the messages, and processes the data in the second cache according to the content of the messages in the second message queue.

[0012] On the other hand, the method further includes: The cloud platform deployed in the edge region receives the second data change event of the second database; wherein, the second database is located in the cloud platform deployed in the edge region; Processes the data in the second cache according to the second data change event.

[0013] On the other hand, there are multiple event listening and parsing engines for monitoring the first database in the cloud platform deployed in the central region, and there are multiple event listening and parsing engines for monitoring the second database in the cloud platform deployed in the edge region; Obtaining the data change event of the database includes: Obtaining the data change event of the database through multiple event listening and parsing engines; Processing the data in the cache according to the data change event includes: Obtaining the main event listening and parsing engine from multiple event listening and parsing engines; Processing the data in the cache according to the data change event obtained by the main event listening and parsing engine.

[0014] On the other hand, the event listening and parsing engine obtaining the data change event of the database includes: The event listening and parsing engine acts as a slave node of the database cluster and starts the input / output thread; Sends a data export request to the master node in the database cluster, so that the master node in the database cluster starts the log export thread and uses the log to respond to the data export request of the event listening and parsing engine; Receives and parses the log to obtain the data change event of the database.

[0015] On the other hand, the method further includes: The cloud platform deployed in the edge region obtains the data to be written into the second cache from the second database; Clear the data in the second cache and store the data to be written into the second cache in the second cache; Obtain the target information from the cloud platform deployed in the central region and store the target information in the second cache; wherein, the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region.

[0016] On the other hand, the method further includes: Obtain the data to be written into the first cache from the first database; Clear the data in the first cache and store the data to be written into the first cache in the first cache.

[0017] On the other hand, clearing the data in the second cache and storing the data to be written into the second cache in the second cache, or clearing the data in the first cache and storing the data to be written into the first cache includes: When it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is a preset moment, or when it is detected that the user issues an instruction for triggering cache reset, clear the data in the second cache and store the data to be written into the second cache in the second cache, or clear the data in the first cache and store the data to be written into the first cache in the first cache.

[0018] On the other hand, after obtaining the data to be written into the first cache from the first database and before clearing the data in the first cache and storing the data to be written into the first cache in the first cache, it further includes: Stop processing the data in the first cache according to the first data change event; After clearing the data in the first cache and storing the data to be written into the first cache in the first cache, it further includes: Process the data in the first cache according to the first data change event; After the cloud platform deployed in the edge region obtains the data to be written into the second cache from the second database and before clearing the data in the second cache and storing the data to be written into the second cache in the second cache, it further includes: Stop processing the data in the second cache according to the second data change event; After obtaining the target information from the cloud platform deployed in the central region and storing the target information in the second cache, it further includes: Process the data in the second cache according to the second data change event.

[0019] On the other hand, after processing the data in the first cache according to the first data change event, it further includes: Obtain the first data change events that have not been successfully processed; Store the first data change events that have not been successfully processed into the message queue for representing processing failures in the first message queue; Start an independent thread to process the first data change events that have not been successfully processed in the message queue for representing processing failures; Delete the cache data corresponding to the first data change events that have not been successfully processed from the first cache; On the other hand, after processing the data in the second cache according to the second data change event, it further includes: Obtain the second data change events that have not been successfully processed; Store the second data change events that have not been successfully processed into the message queue for representing processing failures in the second message queue; Start an independent thread to process the second data change events that have not been successfully processed in the message queue for representing processing failures; Delete the cache data corresponding to the second data change events that have not been successfully processed from the second cache.

[0020] On the other hand, the method further includes: The cloud platform deployed in the edge region receives a request sent by the user for representing querying data from the second cache through a unified interface; if it is detected that the data corresponding to the request exists in the second cache, the data corresponding to the request is returned to the user; if it is detected that the data corresponding to the request does not exist in the second cache, the application programming interface of the cloud platform deployed in the central region is called to obtain the data corresponding to the request; Or, receive a request sent by the user for representing querying data from the first cache through a unified interface; Return the data corresponding to the request existing in the first cache to the user.

[0021] To solve the above technical problems, the present invention further provides a method for processing data consistency of a multi-region cloud platform, which is applied to the cloud platform deployed in the edge region, and the method includes: Receive messages in the second message queue; wherein, the messages in the second message queue are obtained by the cloud platform deployed in the central region from the first data change event of the first database; process the data in the first cache according to the first data change event, and store the first data change event into the first message queue; the message used to represent the first data change event in the first message queue is obtained by cross-region transmission to the second message queue in the cloud platform deployed in the edge region; wherein, the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region; Process the data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0022] To solve the above technical problems, the present invention also provides a multi-region cloud platform data consistency processing device, which is applied to the cloud platform deployed in the central region; including: An acquisition module, configured to acquire the first data change event of the first database; A processing and storage module, configured to process the data in the first cache according to the first data change event, and store the first data change event into the first message queue; wherein, the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region; A transmission module, configured to cross-region transmit the message used to represent the first data change event in the first message queue to the second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region can process the data in the second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0023] To solve the above technical problems, the present invention also provides a computer program product, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the above multi-region cloud platform data consistency processing method are implemented.

[0024] To solve the above technical problems, the present invention also provides a multi-region cloud platform system, including: A memory, configured to store computer programs; A processor, configured to implement the steps of the above multi-region cloud platform data consistency processing method when executing the computer programs.

[0025] To solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above multi-region cloud platform data consistency processing method are implemented.

[0026] The beneficial effects of the present invention are as follows. First, in this method, after detecting a change in the data of the central database, the data transformation is converted into a message in the message queue, and the message is transmitted across domains to the message queue of the edge regional cloud platform. The edge regional cloud platform updates its own cache according to the message in the message queue. This ensures the data consistency between the database of the cloud platform deployed in the central region and the cache of the cloud platform deployed in the edge region. Moreover, after detecting a change in the data of the cloud platform deployed in the central region, the data in the cache of the cloud platform deployed in the central region is updated, which ensures the data consistency between the cache of the cloud platform deployed in the central region and the database of the cloud platform deployed in the central region. That is, through this method, the data consistency among the data in the database, the cache data of the cloud platform deployed in the central region, and the cache data of the cloud platform deployed in the edge region is achieved. Second, compared with the method of achieving data consistency by using the double-write mode, in a multi-region scenario, the present invention does not directly write data to the edge cache, but indirectly writes data to the edge cache through the message queue at the edge. Even if the cache services in each region are not exposed externally for security reasons, the solution of the present invention can still achieve data consistency among multi-region cloud platforms. Third, compared with the method of achieving data consistency by using asynchronous cache updates, in a multi-region scenario, the database of the cloud platform deployed in the central region of the present invention interacts with the message queue of the cloud platform deployed in the edge region through its own message queue, that is, the database of the cloud platform deployed in the central region is indirectly connected to the edge message queue through the central message queue, enabling the database of the cloud platform deployed in the central region to indirectly interact with the edge message queue and achieving data consistency among multi-region cloud platforms. In addition, a set of data located in the cloud platform deployed in the central region is shared among multiple regions. The cloud platform deployed in the edge region does not need to maintain a complete set of data and can obtain data consistent with the cloud platform deployed in the central region only through the cache and the message queue, reducing the deployment and maintenance costs. At the same time, better unified user management is achieved. Moreover, even when there is no data table in the database of the cloud platform deployed in the edge region, data consistency among multi-region cloud platforms is also achieved.

[0027] In addition, only when it is detected that the first data change event is a preset event, the first data change event is stored in the first message queue and then transmitted across domains. When the first data change event is not a preset event, the first data change event is processed locally and no cross-domain transmission is required. In this method, not all messages corresponding to the first data change events are transmitted across domains, reducing the burden of irrelevant data transmission and improving the efficiency of achieving data consistency.

[0028] Configure the parameters of the message forwarding task in the plugin for message forwarding, and use the plugin for message forwarding to forward messages, eliminating the need to write a custom cross-message queue bridge, and enabling smooth expansion to more regions or multi-active data center scenarios. Moreover, the messages are transmitted to cloud platforms deployed in multiple edge regions through the plugin for message forwarding, achieving message synchronization in multiple edge regions.

[0029] The cloud platform deployed in the edge region processes the data in the second cache in a single-threaded or serial manner based on the parsed messages, avoiding the risk of cache inconsistency caused by concurrent write operations. The cloud platform deployed in the edge region stores the unprocessed message events in the second message queue, which is used to represent the message queue for processing failures; and processes the data in the second cache according to the messages in the message queue for processing failures. That is, through retry operations, data eventual consistency is ensured.

[0030] The cloud platform deployed in the edge region processes based on the priority or timestamp of the messages, ensuring the real-time and accuracy of cache updates. By monitoring the database of the cloud platform deployed in the edge region and updating the cache of the cloud platform deployed in the edge region, it is ensured as much as possible that the data in the database of the cloud platform deployed in the edge region also exists in the cache.

[0031] There are multiple event listening parsers that monitor database changes in both the cloud platform deployed in the central region and the cloud platform deployed in the edge region, increasing the probability of successfully capturing database change events; while only updating the cache according to the events captured by the master node during cache update, avoiding the problem of data inconsistency between the database and the cache of the cloud platform deployed in the central region caused by multiple event listening parsers updating the cache.

[0032] As a slave node of the database cluster, the event listening parser, based on the master-slave protocol of the message queue, realizes real-time monitoring of the log changes of the master node in the database cluster and monitors the data change events of the database.

[0033] The cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database; clears the data in the second cache and stores the data to be flushed into the second cache in the second cache; obtains the target information from the cloud platform deployed in the central region and stores the target information in the second cache; where the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region. It realizes the full refresh of the data of the cloud platform deployed in the edge region, can correct the incremental updates that the cloud platform deployed in the edge region may miss during offline or network anomalies, and keeps the cache consistent with the data of the cloud platform deployed in the central region eventually.

[0034] The cloud platform deployed in the central region obtains the data to be flushed into the first cache from the first database; clears the data in the first cache, and stores the data to be flushed into the first cache in the first cache. It realizes the full refresh of the data of the cloud platform deployed in the central region, can correct the incremental updates that the cloud platform deployed in the central region may miss during offline or network anomalies, and keeps the cache of the cloud platform deployed in the central region consistent with the database of the cloud platform deployed in the central region finally.

[0035] When it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is the preset moment, or when it is detected that the user issues an instruction for triggering the reset of the cache, the full refresh of the data of the cloud platform deployed in the central region and the full refresh of the data of the cloud platform deployed in the edge region are performed, ensuring data consistency; and compared with the method of performing full refresh in all scenarios, it reduces the consumption of resources.

[0036] When performing a full refresh, stop processing the data in the cache according to the data change event, that is, stop the real-time update cache operation; after the full refresh is completed, continue the real-time update cache operation to ensure the integrity of the cache data.

[0037] For the data change events that are not processed successfully, store them in the message queue for indicating processing failure, start an independent thread for processing, and delete the corresponding data from the cache to ensure the final data consistency.

[0038] Set a unified interface query in both the cloud platform deployed in the central region and the cloud platform deployed in the edge region. The cloud platform deployed in the edge region or other callers only need to access the unified interface, avoiding frequent cross-region access to the database of the cloud platform deployed in the central region, improving the query performance and simplifying the system design.

[0039] The present invention also provides a method for processing multi-region cloud platform data consistency applied to the cloud platform deployed in the edge region, a device for processing multi-region cloud platform data consistency, a computer program product, a multi-region cloud platform, and a computer-readable storage medium, which have the same or corresponding technical features as the method for processing multi-region cloud platform data consistency applied to the cloud platform deployed in the central region described above, and the effects are the same. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1Schematic diagram of a multi-region cloud platform provided by the present invention; Figure 2 Flowchart of a method for processing data consistency of a multi-region cloud platform provided by the present invention; Figure 3 Schematic diagram of the working principle of an event listening parser provided by an embodiment of the present invention; Figure 4 Flowchart of a data transmission method provided by an embodiment of the present invention; Figure 5 Overall schematic diagram of a data consistency method for a multi-region deployed cloud platform provided by an embodiment of the present invention; Figure 6 Structural diagram of a multi-region cloud platform system provided by an embodiment of the present invention. Detailed implementation manners

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0043] The core of the present invention is to provide a method, device, product and system for processing data consistency of a multi-region cloud platform to solve the technical problem that the related technologies for ensuring data consistency cannot ensure the data consistency of a multi-region cloud platform.

[0044] Figure 1 Schematic diagram of a multi-region cloud platform provided by the present invention, as Figure 1 shown, including a cloud platform deployed in the central region and a cloud platform deployed in the edge region. Both the cloud platform deployed in the central region and the cloud platform deployed in the edge region include their own databases and caches.

[0045] The cloud platform deployed in the central region maintains all data, including user information, resource information, etc., while the cloud platforms deployed in each edge region only deploy necessary functions to provide services. Since the user information is only stored in the database of the cloud platform deployed in the central region and there is no corresponding user record in the database of the cloud platform deployed in the edge region, therefore, in the present invention, the data consistency of the multi-region cloud platform needs to ensure the data consistency of the database of the cloud platform deployed in the central region, the cache of the cloud platform deployed in the central region, and the cache of the cloud platform deployed in the edge region.

[0046] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. A method for processing data consistency in a multi-region cloud platform provided by the present invention is applied to a cloud platform deployed in a central region. Figure 2 The flowchart of a method for processing data consistency in a multi-region cloud platform provided by the present invention is as Figure 2 shown, and the method includes: S10: Obtain a first data change event of a first database; wherein, the first database is located in the cloud platform deployed in the central region; S11: Process the data in the first cache according to the first data change event, and store the first data change event in a first message queue; wherein, both the first cache and the first message queue are located in the cloud platform deployed in the central region; S12: Transmit the message representing the first data change event in the first message queue across domains to a second message queue in the cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0047] The changes in the database include adding, deleting, or updating data. There is no limitation on the method for obtaining the first data change event of the first database, as long as the changes in the database can be monitored. In the present invention, an event listening parser for monitoring the first database is deployed in the cloud platform deployed in the central region. In order to increase the probability of successfully capturing database change events, in practice, multiple event listening parsers for monitoring the first database are deployed in the cloud platform deployed in the central region (i.e., multi-copy deployment). Obtaining the data change event of the database includes: obtaining the data change event of the database through multiple event listening parsers. In addition, in order to avoid the problem of data inconsistency between the database and the cache in the cloud platform deployed in the central region caused by multiple event listening parsers all updating the cache, processing the data in the cache according to the data change event includes: obtaining the main event listening parser from multiple event listening parsers; processing the data in the cache according to the data change event obtained by the main event listening parser. There is no limitation on the selected main event listening parser, which is determined according to the actual situation.

[0048] After obtaining the first data change event, if all the first data change events are transmitted to the cloud platform deployed in the edge region, the amount of data to be transmitted is relatively large. Therefore, before storing the first data change event in the first message queue, it further includes: In the case where it is detected that the first data change event is a preset event, enter the step of storing the first data change event in the first message queue; In the case that the detected first data change event is not a preset event, process the first data change event locally.

[0049] There is no limitation on the preset event. For example, in the database of the cloud platform deployed in the edge area, user data is not stored. Therefore, user data can be obtained from the cloud platform deployed in the central area. At this time, the preset event is an event related to the user.

[0050] The data change events obtained by the event listening parser from the database include: The event listening parser acts as a slave node of the database cluster and starts the input / output threads. Initiate a data export request to the master node in the database cluster, so that the master node in the database cluster starts a log export thread and uses the log to respond to the data export request of the event listening parser. Receive and parse the log to obtain the data change events of the database.

[0051] The event listening parser can determine according to the configuration file that the changes of some tables or fields need to be synchronized across regions, and the changes of other tables are only processed locally in the central region or ignored. The event listening parser supports the configuration adjustment of dynamic tables and fields to meet the data synchronization requirements in different business scenarios, and supports the switching between incremental data push and full data push. Figure 3 It is a schematic diagram of the working principle of an event listening parser provided by an embodiment of the present invention. As Figure 3 shown, it includes the master node of the database cluster and the event listening parser. Here, the event listening parser belongs to the slave node of the database cluster relative to the master node of the database cluster. The working process of the event listening parser includes: 1. Disguise as a slave node: The event listening parser disguises itself as a slave node of the database (MySQL, a relational database management system) cluster, starts the input / output (I / O) threads, and initiates a dump request to the MySQL master node. The master node starts the binlog dump thread and pushes the Binlog log to the I / O thread of this device. In this way, based on the master-slave protocol of MySQL, the event listening parser can monitor the changes of the Binlog log of the master node in the database cluster in real time; 2. Event parsing: For insert, update, and delete events, parse out information such as the table name, field value, and change type; 3. Event filtering: It can be determined according to the configuration how to process the events of different tables, and directly discard the table events that are not concerned, improving the processing efficiency; 4. If the table is user_entity (related to the user), then push the change information to the cloud platform deployed in the edge area; 5. Data that is not in the user_entity table can be locally processed within the cloud platform deployed in the central region (e.g., only update the cache of the cloud platform deployed in the central region or execute other logic), and is not pushed to the cloud platform deployed in the edge region.

[0052] In the process of cross-region transmitting the message in the first message queue that represents the first data change event to the second message queue in the cloud platform deployed in the edge region, a custom cross-message queue bridge can be written, but this process is relatively complex. Therefore, in the present invention, cross-region transmitting the message in the first message queue that represents the first data change event to the second message queue in the cloud platform deployed in the edge region includes: Configure the parameters of the message forwarding task in the plugin for message forwarding; wherein, the parameters of the message forwarding task at least include the first message queue, the second message queue, and the connection information between the first message queue and the second message queue; Connect the plugin for message forwarding to the first message queue according to the parameters of the message forwarding task, pull the message in the first message queue that represents the first data change event, and transmit the message in the first message queue that represents the first data change event to the second message queue in the cloud platform deployed in the edge region.

[0053] At the same time, in order to achieve message synchronization in multiple edge regions, the second message queue in the parameters of the message forwarding task is located in the cloud platforms deployed in multiple edge regions. Transmitting the message in the first message queue that represents the first data change event to the second message queue in the cloud platform deployed in the edge region includes: transmitting the message in the first message queue that represents the first data change event to the second message queue in the cloud platforms deployed in multiple edge regions.

[0054] The cross-region transmission of messages will be further described below. In the original single-region architecture, the message queue only needs to transfer messages within the same data center or the same set of message queue clusters. In a multi-region scenario, the cloud platform deployed in the central region and the cloud platform deployed in the edge region each have independent message queue clusters. In order to enable messages to automatically flow from the message queue of the cloud platform deployed in the central region to the message queue of the cloud platform deployed in the edge region, the Shovel mechanism of the message queue is required. This plugin for message forwarding has the Shovel mechanism. After the event listener parser outputs the change event of user_entity, it is first written into the message queue of the cloud platform deployed in the central region. Then, use this plugin for message forwarding to perform cross-region transmission of messages. The specific principle of this plugin for message forwarding is briefly described as follows: 1) Shovel mechanism: Configure Shovel through the management end of the cloud platform deployed in the central region, with the queue of the cloud platform deployed in the central region as the source and the message queue of the cloud platform deployed in the edge region as the target; 2) The Shovel continuously monitors the source queue. Once a new message is detected, it will automatically "shovel" it to the target message queue of the cloud platform deployed in the edge region. 3) Strategies such as configurable message confirmation, prefetch limit, and network retry can be configured to ensure the reliability of message transmission across the network.

[0055] After the cloud platform deployed in the edge region receives the message from the cloud platform deployed in the central region, it is stored in the local queue waiting to be consumed, ensuring the synchronization of messages across the message queue cluster.

[0056] Through the above process, the automatic forwarding of messages between message queue clusters is achieved, enabling the changes to user_entity in the cloud platform deployed in the central region to be "cross-regionally" delivered to the cloud platform deployed in the edge region.

[0057] After the message is transmitted across domains to the cloud platform deployed in the edge region through the plugin for message forwarding, in the cloud platform deployed in the edge region, the database does not store user data, but still needs to maintain a user cache consistent with the cloud platform deployed in the central region. Therefore, it is necessary to refresh the cache of the cloud platform deployed in the edge region. To avoid the risk of cache inconsistency caused by concurrent write operations, the cloud platform deployed in the edge region processes the data in the second cache according to the messages in the second message queue, including: the cloud platform deployed in the edge region parses the messages in the second message queue; processes the data in the second cache in a single-threaded or serial manner and according to the parsed messages.

[0058] After the cloud platform deployed in the edge region processes the received messages, there may be messages that are not processed successfully, that is, the cache is not updated successfully, resulting in data inconsistency. To achieve data consistency, after the cloud platform deployed in the edge region processes the data in the second cache according to the messages in the second message queue, it also includes: The cloud platform deployed in the edge region starts from processing the data in the second cache according to the messages in the second message queue, obtains unprocessed message events within a preset duration, and stores the unprocessed message events in the second message queue used as the message queue representing processing failure; processes the data in the second cache according to the messages in the message queue representing processing failure.

[0059] The preset duration is not limited and is determined according to the actual situation. In addition, the cloud platform deployed in the edge region processes the data in the second cache according to the messages in the second message queue, including: The cloud platform deployed in the edge region obtains the priority order of the messages in the second message queue and the timestamp of receiving the messages; Combines the priority order of the messages in the second message queue and the timestamp order of receiving the messages and processes the data in the second cache according to the content of the messages in the second message queue.

[0060] The cloud platform deployed in the edge area processes based on the priority or timestamp of messages, ensuring the real-time and accuracy of cache updates.

[0061] The above process can be called the cache refresh of the cloud platform deployed in the edge area. In implementation, a message consumer (such as microservices, iregion, iresource-common) is deployed on the cloud platform deployed in the edge area to consume the user_entity update messages from the message queue of the cloud platform deployed in the edge area. The specific process of cache refresh is as follows: 1. Consume messages: When receiving a message, parse information such as the user primary key identifier (Identity, ID) and change type (insert, update, delete) in the message; 2. Refresh the cache in a single thread: To avoid concurrency issues, perform operations on the cache using a single thread: 1) If it is a deletion event, delete the corresponding cache key; 2) If it is an update / new addition event, update the data in the cache; 3. Failure retry: If writing to the cache fails, write the event to the failure queue of the local message queue (or other custom retry queues), and perform retries or automatic error correction later to ensure eventual consistency.

[0062] Since the cloud platform deployed in the edge area does not have a user_entity data table and cannot directly drive cache updates based on local Binlog, the cross-region message queue event has become the only reliable incremental update source. The cache of the cross-region cloud platform is updated according to the message queue.

[0063] In practice, there may be data changes in the database of the cloud platform deployed in the edge area. To ensure the data consistency between the database of the cloud platform deployed in the edge area and the cache of the cloud platform deployed in the edge area, the data consistency processing method for the multi-region cloud platform also includes: The cloud platform deployed in the edge area receives the second data change event of the second database; where the second database is located in the cloud platform deployed in the edge area; and processes the data in the second cache according to the second data change event.

[0064] The cloud platform deployed in the edge area monitors the changes of the second database in the same way as described above for the cloud platform deployed in the edge area monitoring the changes of the first database, both through the event listening parser for monitoring. In addition, to increase the probability of successfully capturing database change events, there are multiple event listening parsers in the cloud platform deployed in the edge area for monitoring the second database. The principle of how the event listening parser implements monitoring has been described above and will not be elaborated here.

[0065] In practice, for the cloud platforms deployed in the edge regions, there may be situations such as unstable network and offline, resulting in omission of updates to some cached data. In such cases, to ensure data consistency, the data consistency processing method for multi-region cloud platforms further includes: The cloud platform deployed in the edge region obtains the data to be written into the second cache from the second database; Clear the data in the second cache and store the data to be written into the second cache in the second cache; Obtain the target information from the cloud platform deployed in the central region and store the target information in the second cache; wherein, the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region.

[0066] The above process can be referred to as the full refresh (also known as cache reset) of the cache of the cloud platform deployed in the edge region. Figure 4 It is a flowchart of a data transmission method provided by an embodiment of the present invention. As Figure 4 shown, the method includes: S13: The first database enables log listening; S14: The event listening parser parses the event and pushes the event to the message queue of the cloud platform deployed in the central region; S15: The message queue of the cloud platform deployed in the central region shovels the message to the message queue of the cloud platform deployed in the edge region through the shoveling mechanism; S16: After message consumption, refresh the cache of the cloud platform deployed in the edge region; S17: Periodically / start a full refresh to actively obtain data from the message queue of the cloud platform deployed in the central region.

[0067] If full refresh is performed in all scenarios, it will consume a large amount of resources. Therefore, when it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is a preset moment, or when it is detected that the user issues an instruction for triggering cache reset, clear the data in the second cache and store the data to be written into the second cache in the second cache.

[0068] The process of the full refresh of the cloud platform deployed in the edge region described above is described again below: 1. Trigger scenarios: 1) When the system starts; 2) Scheduled tasks (such as every day at midnight); 3) Manual trigger (when there is a large-scale inconsistency or maintenance is required); 2. Execution process: 1) Call the application programming interface (API) of the central region to batch obtain the complete user_entity data; 2) Clear the cache keys related to user_entity in the second cache; 3) Write the full amount of data into the second cache to ensure consistency with the cloud platform deployed in the central region.

[0069] If new incremental changes occur during the full refresh, secondary compensation or replay can be performed after the refresh to ensure the data is up-to-date.

[0070] Through full reset, the incremental updates that may be missed by the cloud platform deployed in the central region during offline or network anomalies can be corrected, so that the cache of the cloud platform deployed in the central region is finally consistent with the database of the cloud platform deployed in the central region.

[0071] The above describes the full refresh of the cache of the cloud platform deployed in the edge region. In practice, the cloud platform deployed in the central region may also experience network instability, offline, etc., resulting in some cache data updates being missed. In this case, to ensure data consistency, the multi-region cloud platform data consistency processing method also includes: Obtain the data to be flushed into the first cache from the first database; Clear the data in the first cache and store the data to be flushed into the first cache in the first cache.

[0072] Similarly, to reduce the consumed resources, when it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is a preset moment, or when it is detected that the user issues an instruction for triggering cache reset, clear the data in the first cache and store the data to be flushed into the first cache in the first cache.

[0073] The above process realizes the full refresh of the second cache of the cloud platform deployed in the edge region and the full refresh of the first cache of the cloud platform deployed in the central region. If the corresponding cache is updated according to the change event of the database (referred to as real-time update cache) during the full refresh process, it may affect the integrity of the data. Therefore, after obtaining the data to be flushed into the first cache from the first database and before clearing the data in the first cache and storing the data to be flushed into the first cache in the first cache, it also includes: Stop processing the data in the first cache according to the first data change event.

[0074] After clearing the data in the first cache and storing the data to be flushed into the first cache in the first cache, it also includes: Process the data in the first cache according to the first data change event.

[0075] That is, when the cache of the cloud platform deployed in the central region is fully refreshed, the task of real-time updating the cache is blocked, and only the full refresh is performed (only the execution process of the timed task is performed). After the full refresh is completed, the real-time cache update operation is continued, ensuring the integrity of the cache data.

[0076] Similarly, to ensure the integrity of the cache data of the cloud platform deployed in the central region, after the cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database, before clearing the data in the second cache and storing the data to be flushed into the second cache into the second cache, it further includes: Stop processing the data in the second cache according to the second data change event.

[0077] After obtaining the target information from the cloud platform deployed in the central region and storing the target information into the second cache, it further includes: Process the data in the second cache according to the second data change event.

[0078] That is, when the cache of the cloud platform deployed in the edge region is fully refreshed, the task of real-time updating the cache is blocked, and only the full refresh is performed (only the execution process of the timed task is performed). After the full refresh is completed, the real-time cache update operation is continued, ensuring the integrity of the cache data.

[0079] In practice, there may be data change events that are not processed successfully. In this case, to ensure data consistency, after processing the data in the first cache according to the first data change event, it further includes: Obtain the first data change event that is not processed successfully; Store the first data change event that is not processed successfully into the message queue for representing processing failure in the first message queue; Start an independent thread to process the first data change event that is not processed successfully in the message queue for representing processing failure; Delete the cache data corresponding to the first data change event that is not processed successfully from the first cache.

[0080] After processing the data in the second cache according to the second data change event, it further includes: Obtain the second data change event that is not processed successfully; Store the second data change event that is not processed successfully into the message queue for representing processing failure in the second message queue; Start an independent thread to process the second data change event that is not processed successfully in the message queue for representing processing failure; Delete the cached data corresponding to the second data change event that has not been successfully processed from the second cache.

[0081] In this method, for the data change events that have not been successfully processed, they are stored in a message queue used to represent processing failures, an independent thread is started for processing, and the corresponding data is deleted from the cache to ensure the final data consistency.

[0082] When a user queries data from a cloud platform deployed in the edge region or from a cloud platform deployed in the central region, in order to improve query performance and simplify system design, the multi-region cloud platform data consistency processing method further includes: The cloud platform deployed in the edge region receives a request sent by the user for querying data from the second cache through a unified interface (such as a Software Development Kit (SDK)); if it is detected that the data corresponding to the request exists in the second cache, the data corresponding to the request is returned to the user; if it is detected that the data corresponding to the request does not exist in the second cache, the application programming interface of the cloud platform deployed in the central region is called to obtain the data corresponding to the request; Or, receive a request sent by the user for querying data from the first cache through a unified interface; Return the data corresponding to the request that exists in the first cache to the user.

[0083] The above process is called the cache query process. For the query of user data by the application service, the cloud platform deployed in the edge region provides a unified SDK for cache query. The caller only needs to provide keyword fields such as the user ID, and the SDK will perform the following steps: 1. First, access the cache of the cloud platform deployed in the edge region; 2. If the cache hits, directly return the data; 3. If it misses, call the interface of the cloud platform deployed in the central region to further confirm whether the data exists and update the cache.

[0084] In a multi-region scenario, if the cloud platform deployed in the edge region does not have the user information in the local database, the query interface of the cloud platform deployed in the central region can be directly called according to business requirements to complete the "penetration" query. In this way, the burden on developers can be further reduced and the system consistency can be improved.

[0085] A multi-region cloud platform data consistency processing method applied to a cloud platform deployed in the central region is described above. This embodiment also provides a multi-region cloud platform data consistency processing method applied to a cloud platform deployed in the edge region. The method includes: Receive messages in the second message queue; where the messages in the second message queue are obtained by the cloud platform deployed in the central region from the first data change events of the first database; process the data in the first cache according to the first data change events, and store the first data change events in the first message queue; the messages in the first message queue representing the first data change events are obtained by cross-region transmission to the second message queue in the cloud platform deployed in the edge region; where the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region. Process the data in the second cache according to the messages in the second message queue; where the second cache is located in the cloud platform deployed in the edge region.

[0086] The method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the edge region provided in this embodiment has the same or corresponding technical features as the method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the central region described above. The embodiments of the method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the central region have been described in detail above, and the embodiments of the method for processing data consistency of a multi-region cloud platform applied to the cloud platform deployed in the edge region will not be elaborated here, and the effects are the same.

[0087] To enable those skilled in the art to better understand the overall solution of the data consistency method of the multi-region deployed cloud platform of the present invention, the following will continue to be described in conjunction with the accompanying drawings and specific embodiments. Figure 5 It is a schematic diagram of the overall data consistency method of a multi-region deployed cloud platform provided in an embodiment of the present invention.

[0088] In the cloud platform deployed in the central region: 1.1. Monitor the log events of the first database and discard the events that do not need to be flushed into the cache. 1.2. The master node in the multi-copy scenario updates the cache in real time.

[0089] The above steps 1.1 to 1.2 are called the steps of updating the cache in real time.

[0090] 1.3. The events of cache write failure are temporarily stored in the message queue. 1.4. Start an independent thread to consume the events and delete the cache data corresponding to the write failure events. 1.5. Monitor the user-related log events of the first database.

[0091] Push the monitored user-related log events to the first message queue through the microservice (iregion); transfer them across domains to the second message queue in the cloud platform deployed in the edge region through the shovel mechanism, and input them into the second cache through microservice one (iregion) and microservice two (iresource-common).

[0092] 2.1. Read the data that needs to be flushed into the cache from the first database; 2.2. Clear the cache and flush the read data into the cache.

[0093] Steps 2.1 to 2.2 can be called the full refresh task of the cache in the cloud platform deployed in the central region. It should be noted that during the execution of the full refresh task, the real-time update of the cache is blocked.

[0094] In the cloud platform deployed in the edge region: 1.1. Monitor the log events of the second database and discard the events that do not need to be flushed into the cache; 1.2. The master node in the multi-copy scenario updates the cache in real time.

[0095] The above steps 1.1 to 1.2 are called the steps for real-time cache update.

[0096] 1.3. Cache write failure events are temporarily stored in the message queue; 1.4. Start an independent thread to consume events and delete the cache data corresponding to the write failure events. The above steps 1.1 to 1.4 are called the steps for real-time cache update; 2.1. Read the data that needs to be flushed into the cache from the second database; 2.2. Clear the cache and flush the read data into the cache; 2.3. Clear the user cache, query the user information in the central region, and flush it into the cache.

[0097] Steps 2.1 to 2.3 can be called the full refresh task of the cache in the cloud platform deployed in the edge region. It should be noted that during the execution of the full refresh task, the real-time update of the cache is blocked.

[0098] The overall idea is as follows: 1. The database (i.e., the first database) in the cloud platform deployed in the central region enables the Binlog function to record all data changes.

[0099] 2. Event listening parser: Deploy a listening program in the cloud platform deployed in the central region to parse Binlog events in real time and obtain information such as table names, primary keys, and change types.

[0100] 1) Different processing strategies can be adopted for different tables; 2) If it is a user_entity table event related to cross-region synchronization, push a message to the edge Region; 3) Data in other tables is only processed locally (not pushed) on the cloud platform deployed in the central region.

[0101] 3. Cross-region message distribution: Write the captured user_entity table events into the first message queue of the cloud platform deployed in the central region, and then use the shovel mechanism of the first message queue to "shovel" the messages to the second message queue of the cloud platforms deployed in each edge region.

[0102] 4. Cache refresh of the cloud platform deployed in the edge region: The cloud platform deployed in the edge region listens to the local second message queue. After receiving the incremental update message from the cloud platform deployed in the central region, update / delete the corresponding data in the second cache.

[0103] 5. Full cache refresh of the edge Region: When the cache of the cloud platform deployed in the edge region is scheduled or at startup, actively pull the complete user_entity data from the cloud platform deployed in the central region, empty the user-related cache in the second cache and then reload it to correct possible omissions or deviations.

[0104] 6. Cache queryer: When the application queries user information on the cloud platform deployed in the edge region, it can first read the cache; if the cache is not hit, call the user information API of the cloud platform deployed in the central region.

[0105] Through this method, the goal of keeping the user data of the cloud platform deployed in the edge region consistent with that of the cloud platform deployed in the central region without a database table is achieved. Specifically: 1. The cloud platform deployed in the central region listens to the Binlog: Parse the real-time add, delete, and modify logs of the database; When it detects that the table name is user_entity, wrap the message body (primary key, change type, field data) and write it into the user_entity_update queue in the first message queue of the cloud platform deployed in the central region.

[0106] 2. Configuration of the shovel mechanism of the first message queue: Set up Shovel on the cloud platform deployed in the central region. The source end is the user_entity_update queue, and the target end is the user_entity_edge queue of the edge Region; When Shovel detects a new message, it automatically grabs and forwards it to the message queue of the cloud platform deployed in the edge region without manual intervention.

[0107] 3. The cloud platform deployed in the edge region receives and refreshes the cache: The local consumer subscribes to the user_entity_edge queue and receives incremental update messages; According to the event type, an insert, update or delete operation is performed on the second cache; If the operation fails, it is recorded in the retry queue or log and can be restored later.

[0108] 4. Full refresh of cloud platforms deployed in edge regions: The scheduled task is triggered every morning, or the following steps are executed when the system starts: 1) Call the API interface of the cloud platform deployed in the central region to obtain all user information; 2) Clear the cache of user_entity in the second cache; 3) Write the data into the second cache one by one; 4) If incremental events arrive during the refresh, they can be replayed again after the refresh is complete to avoid overwriting the updates.

[0109] Through the above process, the cloud platform deployed in the edge region can obtain user data changes in a timely manner and maintain the uniformity of the cache; even if message loss occurs or the edge region goes offline, it can quickly recover to the correct state through a full reset. Multiple regions share a user system, and the edge region does not need to maintain a complete set of user data. It can obtain user data consistent with the central region only through cache and message queues, reducing deployment and maintenance costs, while achieving better unified user management; real-time increment + regular full: Real-time updates are achieved through Binlog monitoring + cross-region message push, and full refresh is performed regularly or at startup to prevent omissions, greatly improving system availability and fault tolerance; the shovel mechanism of the message queue synchronizes across clusters: There is no need to write a custom cross-message queue bridge, and the shovel mechanism provides safe and stable message "shoveling", which can be smoothly expanded to more regions or multi-active data center scenarios; reduce the complexity of application development: the edge region or other callers only need to access the cache query (or unified SDK), avoiding frequent cross-region access to the central database, improving query performance and simplifying system design.

[0110] In the above embodiments, a data consistency method for a multi-regional cloud platform is described in detail. The present invention also provides an embodiment corresponding to a data consistency device for a multi-regional cloud platform.

[0111] An embodiment of the present invention provides a multi-region cloud platform data consistency processing device, which is applied to a cloud platform deployed in a central region; comprising: An acquisition module, used for acquiring a first data change event of a first database; A processing and storage module, configured to process data in the first cache according to a first data change event, and store the first data change event into a first message queue; wherein, the first database, the first cache, and the first message queue are all deployed in a cloud platform in the central region; A transmission module, configured to transmit, across domains, a message representing the first data change event in the first message queue to a second message queue in a cloud platform deployed in the edge region, so that the cloud platform deployed in the edge region processes data in the second cache according to the message in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0112] In some embodiments, the multi-region cloud platform data consistency processing apparatus further includes: A detection and triggering module, configured to trigger the storage module in the processing and storage module when it is detected that the first data change event is a preset event; wherein, the storage module is configured to store the first data change event into the first message queue; A processing module, configured to locally process the first data change event when it is detected that the first data change event is not a preset event.

[0113] In some embodiments, the multi-region cloud platform data consistency processing apparatus further includes a first processing module, configured to process data in the second cache in the cloud platform deployed in the edge region according to the message in the second message queue.

[0114] The first processing module includes: A parsing and processing module, configured to parse the message in the second message queue; and process the data in the second cache in a single-threaded or serial manner according to the parsed message; The multi-region cloud platform data consistency processing apparatus further includes: A first storage module, configured to, in the cloud platform deployed in the edge region, starting from the time when the data in the second cache is processed according to the message in the second message queue, obtain unprocessed message events within a preset duration, and store the unprocessed message events into a message queue for representing processing failures in the second message queue; and process the data in the second cache according to the message in the message queue for representing processing failures.

[0115] The first processing module includes: A first obtaining module, configured to obtain the priority order of the messages in the second message queue and the timestamps of receiving the messages in the cloud platform deployed in the edge region; A second processing module, configured to process the data in the second cache according to the content of the message in the second message queue by combining the priority order of the messages in the second message queue and the timestamp order of receiving the messages.

[0116] In some embodiments, the multi-region cloud platform data consistency processing device further includes: A first receiving module, configured to receive, by the cloud platform deployed in the edge region, a second data change event of a second database; wherein, the second database is located in the cloud platform deployed in the edge region; A third processing module, configured to process the data in the second cache according to the second data change event.

[0117] In some embodiments, the multi-region cloud platform data consistency processing device includes a second obtaining module, configured to obtain a data change event of a database.

[0118] The second obtaining module is specifically configured to obtain a data change event of a database through a plurality of event listening parsers.

[0119] In some embodiments, the multi-region cloud platform data consistency processing device includes a third obtaining module, configured to obtain, by an event listening parser, a data change event of a database. The third obtaining module specifically includes: An enabling module, configured to enable the event listening parser as a slave node of the database cluster and start an input / output thread; A request module, configured to initiate a data export request to a master node in the database cluster, so that the master node in the database cluster starts a log export thread and uses a log to respond to the data export request of the event listening parser; A second receiving module, configured to receive and parse the log to obtain a data change event of a database.

[0120] In some embodiments, the multi-region cloud platform data consistency processing device further includes: A third obtaining module, configured to obtain, by the cloud platform deployed in the edge region, data to be flushed into the second cache from a second database; A first clearing and storing module, configured to clear the data in the second cache and store the data to be flushed into the second cache in the second cache; An obtaining and storing module, configured to obtain target information from the cloud platform deployed in the central region and store the target information in the second cache; wherein, the target information is located in the cloud platform deployed in the central region and not in the cloud platform deployed in the edge region.

[0121] In some embodiments, the multi-region cloud platform data consistency processing device further includes: A fourth obtaining module, configured to obtain data to be flushed into the first cache from a first database; A first clearing and storing module, configured to clear the data in the first cache and store the data to be flushed into the first cache in the first cache.

[0122] In some embodiments, the multi-region cloud platform data consistency processing device further includes: The first stop module is used to stop processing the data in the first cache according to the first data change event.

[0123] In some embodiments, the multi-region cloud platform data consistency processing device further includes: The second stop module is used to stop processing the data in the second cache according to the second data change event.

[0124] In some embodiments, the multi-region cloud platform data consistency processing device further includes: The fifth acquisition module is used to acquire the first data change event that has not been successfully processed; The second storage module is used to store the first data change event that has not been successfully processed into the message queue for representing processing failure in the first message queue; The fourth processing module is used to start an independent thread to process the first data change event that has not been successfully processed in the message queue for representing processing failure; The first deletion module is used to delete the cache data corresponding to the first data change event that has not been successfully processed from the first cache.

[0125] In some embodiments, the multi-region cloud platform data consistency processing device further includes: The sixth acquisition module is used to acquire the second data change event that has not been successfully processed; The third storage module is used to store the second data change event that has not been successfully processed into the message queue for representing processing failure in the second message queue; The fifth processing module is used to start an independent thread to process the second data change event that has not been successfully processed in the message queue for representing processing failure; The second deletion module is used to delete the cache data corresponding to the second data change event that has not been successfully processed from the second cache.

[0126] In some embodiments, the multi-region cloud platform data consistency processing device further includes: The first request processing module is used for the cloud platform deployed in the edge region to receive a request issued by the user for querying data from the second cache through a unified interface; if it is detected that the data corresponding to the request exists in the second cache, the data corresponding to the request is returned to the user; if it is detected that the data corresponding to the request does not exist in the second cache, the application programming interface of the cloud platform deployed in the central region is called to obtain the data corresponding to the request; Or, the second request processing module is used to receive a request issued by the user for querying data from the first cache through a unified interface; and return the data corresponding to the request existing in the first cache to the user.

[0127] An embodiment of the present invention further provides a multi-region cloud platform data consistency processing device applied to a cloud platform deployed in an edge region. The device includes: A third receiving module, configured to receive messages in a second message queue; wherein, the messages in the second message queue are obtained by a cloud platform deployed in a central region from a first data change event of a first database; processing data in a first cache according to the first data change event, and storing the first data change event into a first message queue; and obtaining the messages in the first message queue for representing the first data change event by cross-region transmission to the second message queue in the cloud platform deployed in the edge region; wherein, the first database, the first cache, and the first message queue are all located in the cloud platform deployed in the central region; A sixth processing module, configured to process data in a second cache according to the messages in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge region.

[0128] Since the embodiments in the device part correspond to the embodiments in the method part, for the embodiments in the device part, please refer to the description of the embodiments in the method part, and will not be elaborated here.

[0129] Figure 6 It is a structural diagram of a multi-region cloud platform system provided by an embodiment of the present invention. This embodiment is from a hardware perspective. As Figure 6 shown, the multi-region cloud platform system includes: A memory 20, configured to store a computer program; A processor 21, configured to implement the steps of the multi-region cloud platform data consistency processing method as mentioned in the above embodiments when executing the computer program.

[0130] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array. The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may further include an artificial intelligence (AI) processor, and the AI processor is used to process computational operations related to machine learning.

[0131] The memory 20 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 20 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201. After the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the multi-region cloud platform data consistency processing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may further include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the multi-region cloud platform data consistency processing method mentioned above.

[0132] In some embodiments, the multi-region cloud platform system may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0133] Those skilled in the art can understand that Figure 6 the structure shown in

[0134] The multi-region cloud platform system provided by the embodiments of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, the following method can be implemented: a method for processing data consistency of the multi-region cloud platform, with the same effect.

[0135] The embodiments of the present invention further provide a computer program product, including a computer program / instructions. When the computer program / instructions are executed by the processor, the steps of the above-mentioned method for processing data consistency of the multi-region cloud platform are implemented.

[0136] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps recorded in the above method embodiments (which can be the method corresponding to the cloud platform deployed in the central region, or the method corresponding to the cloud platform deployed in the edge region, or the method corresponding to both the cloud platform side deployed in the central region and the cloud platform side deployed in the edge region) are implemented.

[0137] It can be understood that if the method in the above embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage media include: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs, etc., which can store program codes.

[0138] The computer-readable storage medium provided by the present invention includes the above-mentioned method for processing data consistency of the multi-region cloud platform, with the same effect.

[0139] The above has introduced in detail the method, device, product, and system for processing data consistency of the multi-region cloud platform provided by the present invention. The various embodiments in the specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple. For the relevant parts, refer to the description of the method part. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0140] It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

Claims

1. A multi-region cloud platform data consistency processing method, characterized in that: A cloud platform applied to central regional deployment; the method comprises: Obtaining a first data change event of a first database; Processing data in a first cache according to the first data change event, and storing the first data change event in a first message queue; wherein the first database, the first cache, and the first message queue are all located on a cloud platform deployed in a central region; The message in the first message queue used to characterize the first data change event is transmitted across domains to the second message queue in the cloud platform deployed in the edge area, so that the cloud platform deployed in the edge area can process the data in the second cache according to the message in the second message queue; wherein the second cache is located in the cloud platform deployed in the edge area.

2. The multi-region cloud platform data consistency processing method according to claim 1 is characterized in that: Before storing the first data change event in the first message queue, the method further includes: In the case where it is detected that the first data change event is a preset event, entering the step of storing the first data change event in a first message queue; When it is detected that the first data change event is not a preset event, the first data change event is processed locally.

3. The multi-region cloud platform data consistency processing method according to claim 1 is characterized in that: The cross-domain transmission of the message in the first message queue used to represent the first data change event to the second message queue in the cloud platform deployed in the edge region includes: Configuring parameters of a message forwarding task in a plug-in for message forwarding; wherein the parameters of the message forwarding task include at least the first message queue, the second message queue, and connection information between the first message queue and the second message queue; According to the parameters of the message forwarding task, the plug-in for message forwarding is connected to the first message queue, and the message for representing the first data change event is pulled from the first message queue, and the message for representing the first data change event is transmitted to the second message queue in the cloud platform deployed in the edge area.

4. The multi-region cloud platform data consistency processing method according to claim 3 is characterized in that: The second message queue in the parameters of the message forwarding task is located on a cloud platform deployed in multiple edge regions; The transmitting the message representing the first data change event to the second message queue in the cloud platform deployed in the edge region includes: The message representing the first data change event is transmitted to a second message queue in a cloud platform deployed in multiple edge regions.

5. The multi-region cloud platform data consistency processing method according to claim 1 is characterized in that: The cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, including: The cloud platform deployed in the edge region parses the messages in the second message queue; processes the data in the second cache in a single-threaded or serial manner according to the messages obtained after parsing; After the cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, the method further includes: The cloud platform deployed in the edge area starts by processing the data in the second cache according to the messages in the second message queue, obtains unprocessed message events within a preset time period, and stores the unprocessed message events in the message queue in the second message queue for characterizing processing failure; and processes the data in the second cache according to the messages in the message queue for characterizing processing failure.

6. The multi-region cloud platform data consistency processing method according to claim 1 is characterized in that: The cloud platform deployed in the edge region processes the data in the second cache according to the message in the second message queue, including: The cloud platform deployed in the edge region obtains the priority order of the messages in the second message queue and the timestamp of receiving the messages; The data in the second cache is processed according to the content of the messages in the second message queue in combination with the priority order of the messages in the second message queue and the timestamp order of the received messages.

7. The multi-region cloud platform data consistency processing method according to claim 1 is characterized in that: The method further comprises: The cloud platform deployed in the edge region receives a second data change event of the second database; wherein the second database is located in the cloud platform deployed in the edge region; The data in the second cache is processed according to the second data change event.

8. The multi-region cloud platform data consistency processing method according to claim 7 is characterized in that: There are multiple event listeners and parsers for monitoring the first database in the cloud platform deployed in the central area, and there are multiple event listeners and parsers for monitoring the second database in the cloud platform deployed in the edge area; The data change events for obtaining database include: Obtain database data change events through multiple event listener parsers; Processing cached data based on data change events includes: Get the main event listener parser from multiple event listener parsers; The data in the cache is processed according to the data change event acquired by the main event listener parser.

9. The multi-region cloud platform data consistency processing method according to claim 8 is characterized in that: The event listener parser obtains the data change events of the database including: The event listener parser acts as a slave node of the database cluster and starts the input and output threads; Initiate a data export request to the master node in the database cluster so that the master node in the database cluster starts a log export thread and uses the log response event to listen to the data export request of the parser; Receive and parse logs to obtain database data change events.

10. The multi-region cloud platform data consistency processing method according to claim 7, characterized in that: The method further comprises: The cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database; Clear the data in the second cache, and store the data to be flushed into the second cache in the second cache; Obtain target information from a cloud platform deployed in a central area, and store the target information in the second cache; wherein the target information is located on the cloud platform deployed in the central area, and is not located on the cloud platform deployed in an edge area.

11. The multi-region cloud platform data consistency processing method according to claim 10, characterized in that: The method further comprises: Acquire data to be flushed into the first cache from the first database; The data in the first cache is cleared, and the data to be flushed into the first cache is stored in the first cache.

12. The multi-region cloud platform data consistency processing method according to claim 11, characterized in that: The clearing of data in the second cache and storing the data to be flushed into the second cache in the second cache, or the clearing of data in the first cache and storing the data to be flushed into the first cache in the first cache includes: When it is detected that the multi-region cloud platform is started, or when it is detected that the current moment is a preset moment, or when it is detected that the user issues an instruction for triggering a cache reset, the data in the second cache is cleared, and the data to be flushed into the second cache is stored in the second cache, or the data in the first cache is cleared, and the data to be flushed into the first cache is stored in the first cache.

13. The multi-region cloud platform data consistency processing method according to claim 11, characterized in that: After acquiring the data to be flushed into the first cache from the first database, before clearing the data in the first cache and storing the data to be flushed into the first cache into the first cache, the method further includes: Stop processing the data in the first cache according to the first data change event; After clearing the data in the first cache and storing the data to be flushed into the first cache into the first cache, the method further includes: Processing data in the first cache according to the first data change event; After the cloud platform deployed in the edge region obtains the data to be flushed into the second cache from the second database, before clearing the data in the second cache and storing the data to be flushed into the second cache in the second cache, the method further includes: Stop processing the data in the second cache according to the second data change event; After acquiring target information from a cloud platform deployed in a central region and storing the target information in the second cache, the method further includes: The data in the second cache is processed according to the second data change event.

14. The multi-region cloud platform data consistency processing method according to claim 7, characterized in that: After processing the data in the first cache according to the first data change event, the method further includes: Get the first data change event that was not successfully processed; storing the first data change event that was not successfully processed into a message queue in the first message queue for indicating a processing failure; Initiate an independent thread to process a first data change event that has not been successfully processed in a message queue for indicating a processing failure; deleting cache data corresponding to the first data change event that was not successfully processed from the first cache; After processing the data in the second cache according to the second data change event, the method further includes: Obtain the second data change event that was not successfully processed; storing the unsuccessfully processed second data change event in the second message queue as a message queue for indicating a processing failure; Initiate an independent thread to process a second data change event that has not been successfully processed in a message queue for indicating a processing failure; The cache data corresponding to the second data change event that has not been successfully processed is deleted from the second cache.

15. The multi-region cloud platform data consistency processing method according to any one of claims 1 to 14, characterized in that: The method further comprises: The cloud platform deployed in the edge region receives a request sent by a user through a unified interface for querying data from the second cache; if it is detected that the data corresponding to the request exists in the second cache, the data corresponding to the request is returned to the user; if it is detected that the data corresponding to the request does not exist in the second cache, the application programming interface of the cloud platform deployed in the central region is called to obtain the data corresponding to the request; Or, receiving, through a unified interface, a request sent by a user for representing querying data from the first cache; The data corresponding to the request existing in the first cache is returned to the user.

16. A multi-region cloud platform data consistency processing method, characterized in that: A cloud platform applied to edge region deployment, the method comprising: Receive a message in a second message queue; wherein the message in the second message queue is obtained by a cloud platform deployed in a central region to obtain a first data change event of a first database; process the data in the first cache according to the first data change event, and store the first data change event in the first message queue; and transmit the message in the first message queue used to characterize the first data change event across domains to a second message queue in a cloud platform deployed in an edge region; wherein the first database, the first cache, and the first message queue are all located in a cloud platform deployed in a central region; Process data in a second cache according to messages in a second message queue; wherein the second cache is located on a cloud platform deployed in an edge region.

17. A multi-region cloud platform data consistency processing device, characterized in that: Cloud platform for central regional deployment; including: An acquisition module, used for acquiring a first data change event of a first database; a processing and storage module, configured to process the data in the first cache according to the first data change event, and store the first data change event in a first message queue; wherein the first database, the first cache, and the first message queue are all located on a cloud platform deployed in a central region; A transmission module is used to transmit the message in the first message queue used to represent the first data change event across domains to the second message queue in the cloud platform deployed in the edge area, so that the cloud platform deployed in the edge area can process the data in the second cache according to the message in the second message queue; wherein, the second cache is located in the cloud platform deployed in the edge area.

18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the multi-region cloud platform data consistency processing method described in any one of claims 1 to 16 are implemented.

19. A multi-region cloud platform system, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the multi-region cloud platform data consistency processing method as described in any one of claims 1 to 16 when executing the computer program.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the multi-region cloud platform data consistency processing method as described in any one of claims 1 to 16 are implemented.

Images