Data processing method and device based on message middleware

By pre-setting the producer replicator of the second cluster in the first cluster and using the consumer cursor mechanism of the message middleware, the problem of cluster message replication performance loss is solved, and efficient message replication and disaster recovery capabilities are achieved.

CN114513513BActive Publication Date: 2024-11-19HUNAN HAPPLY SUNSHINE INTERACTIVE ENTERTAINMENT MEDIA CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210138474.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-15
Publication Date
2024-11-19
Estimated Expiration
2042-02-15

AI Technical Summary

Technical Problem

The existing cluster message replication method requires additional machine resources and results in performance losses, affecting the cluster's high availability and disaster recovery capabilities.

Method used

By pre-installing a replicator containing the producer of the second cluster in the first cluster, using the consumption cursor mechanism of the message middleware, the message is copied from the first cluster to the second cluster without affecting the performance of the first cluster. Replication of message is performed to eliminate copy loss across processes and across networks.

Benefits of technology

Without consuming the machine resources of the first cluster, the disaster recovery capability of the cluster is improved and the message replication process is accelerated, ensuring that the performance of the second cluster is not affected.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114513513B_ABST
    Figure CN114513513B_ABST
Patent Text Reader

Abstract

The present application discloses a data processing method and device based on message middleware, the method comprising: saving the message produced by the producer of the first cluster to the target topic of the first cluster; obtaining the message from the message queue corresponding to the target topic based on the consumption cursor of the first replicator, and saving the message to the first replicator; calling the producer in the first replicator, sending the message in the first replicator to the second cluster, so that the target topic of the second cluster saves the message; after determining that the target topic of the second cluster successfully saves the message, sending a confirmation character to the first replicator, so that the first replicator updates the consumption cursor. The method is based on a first replicator that is preset in the first cluster (specifically built into the broker node process) and includes the producer of the second cluster. It can copy the message of the first cluster to the second cluster without consuming the machine resources of the first cluster, thereby improving the disaster recovery capability of the cluster.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of message middleware, and in particular to a data processing method and device based on message middleware. Background Art

[0002] Message middleware is often used in the entire R&D system to perform high-concurrency peak-shaving and valley-filling, and decouple services from each other. It is a very core and indispensable part. The high availability of message middleware is its core capability, and the reliability and availability of message middleware clusters (referred to as clusters) in the same region can be guaranteed by multiple copies and multiple nodes. When clusters in individual regions are abnormal (such as power outages, network outages, and fiber optic failures), consumers in the abnormal cluster need to consume from other clusters across regions (i.e., cross-regional consumption behavior). For clusters that undertake cross-regional consumption behavior, messages in the abnormal cluster need to be copied to themselves in advance.

[0003] Currently, the existing cluster message replication method is usually to use an independent synchronization tool service to send the messages contained in the cluster to another cluster. The existing cluster message replication method requires additional machine resources to deploy the synchronization tool service, and also causes performance loss due to additional cross-process and cross-network data replication, which seriously affects the performance of the cluster and reduces the cluster's disaster recovery capabilities. Summary of the invention

[0004] The present application provides a data processing method and device based on message middleware, which is used to copy messages in a cluster to other clusters without affecting the performance of the cluster, thereby improving the disaster recovery capability of the cluster.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] A data processing method based on message middleware, comprising:

[0007] Saving the message produced by the producer of the first cluster to the target topic of the first cluster; the message includes a data item; the target topic is a topic corresponding to the business type to which the data item belongs;

[0008] Based on the consumption cursor of the first replicator, the message is obtained from the message queue corresponding to the target topic of the first cluster, and the message is saved in the first replicator; the first replicator is preset in the first cluster, and the first replicator includes the producer of the second cluster; the first cluster and the second cluster belong to different regions respectively; the consumption cursor of the first replicator is pre-established based on the first replicator as a consumer of the first cluster;

[0009] Calling the producer in the first replicator to send the message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the message;

[0010] After determining that the target topic of the second cluster successfully saves the message, a confirmation character is sent to the first replicator, so that the first replicator updates its own consumption cursor.

[0011] Optionally, also include:

[0012] In the event of a failure of the first cluster, messages produced by producers of the first cluster are saved in a target topic of the second cluster.

[0013] Optionally, also include:

[0014] When the second cluster is not faulty, save the message produced by the producer of the second cluster to the target topic of the second cluster;

[0015] Based on the consumption cursor of the second replicator, the message is obtained from the message queue corresponding to the target topic of the second cluster, and the message is saved in the second replicator; the second replicator is preset in the second cluster, and the second replicator includes the producer of the first cluster; the consumption cursor of the second replicator is pre-established based on the second replicator as a consumer of the second cluster;

[0016] Calling the producer in the second replicator to send the message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the message;

[0017] After determining that the target topic of the first cluster successfully saves the message, a confirmation character is sent to the second replicator to enable the second replicator to update its own consumption cursor.

[0018] Optionally, the message further includes metadata of the data item, and the metadata includes an ID of the message;

[0019] The step of saving the message produced by the producer of the second cluster to the target topic of the second cluster includes:

[0020] When the value of the ID of the message produced by the producer of the second cluster is not greater than the value of the ID of the message indicated by the consumer cursor of the second replicator, the message is saved in the target topic of the second cluster.

[0021] Optionally, also include:

[0022] When the value of the ID of the message produced by the producer of the second cluster is greater than the value of the ID of the message indicated by the consumer cursor of the second replicator, it is prohibited to save the message in the second cluster and the message is deleted.

[0023] Optionally, the message further includes metadata of the data item, wherein the metadata includes a cluster identifier of a cluster to which a producer of the data item belongs;

[0024] The calling of the producer in the second replicator to send the message stored in the second replicator to the first cluster so that the target topic of the first cluster stores the message includes:

[0025] When the message stored in the second replicator does not include the cluster identifier of the first cluster, the producer in the second replicator is called to send the message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the message.

[0026] Optionally, also include:

[0027] In a case where the message stored in the second replicator includes the cluster identifier of the first cluster, sending the message stored in the second replicator to the first cluster is prohibited.

[0028] A data processing device based on message middleware, comprising:

[0029] A message storage unit, used to store the message produced by the producer of the first cluster in the target topic of the first cluster; the message includes a data item; the target topic is a topic corresponding to the business type to which the data item belongs;

[0030] a message consumption unit, configured to obtain the message from a message queue corresponding to a target topic of the first cluster based on a consumption cursor of the first replicator, and save the message to the first replicator; the first replicator is preset in the first cluster, and the first replicator includes a producer of the second cluster; the first cluster and the second cluster belong to different regions respectively; the consumption cursor of the first replicator is pre-established based on the first replicator being a consumer of the first cluster;

[0031] A message replication unit, configured to call the producer in the first replicator to send the message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the message;

[0032] The cursor updating unit is used to send a confirmation character to the first replicator after determining that the target topic of the second cluster successfully saves the message, so that the first replicator updates its own consumption cursor.

[0033] A computer-readable storage medium includes a stored program, wherein the program executes the data processing method based on message middleware.

[0034] A data processing device based on message middleware, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0035] The memory is used to store programs, and the processor is used to run programs, wherein the data processing method based on message middleware is executed when the program is running.

[0036] The technical solution provided by the present application saves the message produced by the producer of the first cluster to the target topic of the first cluster. Based on the consumption cursor of the first replicator, the message is obtained from the message queue corresponding to the target topic of the first cluster, and the message is saved in the first replicator. The producer in the first replicator is called to send the message saved in the first replicator to the second cluster so that the target topic of the second cluster saves the message. After determining that the target topic of the second cluster successfully saves the message, a confirmation character is sent to the first replicator so that the first replicator updates its own consumption cursor. Compared with the prior art, the solution shown in the present application is based on the first replicator preset in the first cluster and containing the producer of the second cluster. The message contained in the first cluster can be copied to the second cluster without consuming the machine resources of the first cluster, thereby copying the message in the cluster to other clusters without affecting the performance of the cluster, effectively improving the disaster recovery capability of the cluster. In addition, because the first replicator can be built into the broker node process of the first cluster, it can also eliminate the cross-process and cross-network replication loss between data, and improve the speed of copying the message of the first cluster to the second cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1a A flowchart of a data processing method based on message middleware provided in an embodiment of the present application;

[0039] Figure 1b A schematic diagram of a data processing scenario provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of another data processing method based on message middleware provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the architecture of a data processing device based on message middleware provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] like Figure 1a As shown, it is a flow chart of a data processing method based on message middleware provided in an embodiment of the present application, comprising the following steps:

[0044] S101: Based on the first replicator being a consumer of the first cluster, a consumption cursor of the first replicator is pre-established.

[0045] The first replicator is preset in the first cluster, and the first replicator includes the producer of the second cluster, and the first cluster and the second cluster belong to different regions respectively. The consumption cursor of the first replicator is used to indicate the ID of the message that can be consumed by the first replicator in the message queue shown in the first cluster. Generally speaking, the first replicator can be built into the broker node process of the first cluster. The so-called broker node process is common knowledge familiar to those skilled in the art and will not be repeated here.

[0046] Since the first cluster is a message middleware, based on the characteristics of the message middleware, it can be known that: as long as the first replicator subscribes to each topic in the first cluster, the first cluster will push the messages that can be consumed by the first replicator in the message queue corresponding to each topic (in this embodiment of the application, the topic can be equated with the message queue) to the first replicator according to the consumption cursor of the first replicator.

[0047] It should be noted that the number of first replicators included in the first cluster is the same as the number of topics in the first cluster. Of course, the number of first replicators included in the first cluster is the same as the number of topics in the second cluster. In other words, the number of topics included in the first cluster is the same as the number of topics included in the second cluster. In the embodiment of the present application, the topic names included in the first cluster and the second cluster are also the same. For example, the first cluster includes a user name topic, and the second cluster also includes a user name topic.

[0048] S102: When the first cluster is not faulty, the message produced by the producer of the first cluster is saved in the target topic of the first cluster.

[0049] The message includes a data item and metadata of the data item, and the metadata includes at least the message ID and the cluster identifier of the cluster to which the producer of the data item belongs. Specifically, the message ID is determined based on the name of the producer of the data item and the unique incremental ID of the data item, ensuring that the ID of each message is different. In an embodiment of the present application, the target topic is a topic corresponding to the business type to which the data item belongs.

[0050] Based on the characteristics of the message middleware, before the producer sends a message to the first cluster, the producer will carry the verification data schema to the first cluster for compatibility verification and connection establishment. After the producer establishes a connection with the first cluster, the first cluster will perform distributed storage on the message sent by the producer (that is, save the message to the preset distributed storage) and notify the producer that the message was sent successfully.

[0051] In addition, the consumers of the first cluster will establish a connection with the first cluster by verifying the received data schema and setting their own consumption cursor. Based on the characteristics of the message middleware, the first cluster will obtain the messages that can be consumed by consumers from the message queue corresponding to the target topic according to the consumption cursor of the consumers who subscribe to the target topic, and push the messages that can be consumed by consumers to consumers.

[0052] S103: Based on the consumption cursor of the first replicator, obtain a message from the message queue corresponding to the target topic of the first cluster, and save the message to the first replicator.

[0053] S104: Call the producer in the first replicator to send the message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the message.

[0054] Among them, the producer of the second cluster contained in the first replicator can not only synchronously send the messages saved in the first replicator to the second cluster, but also asynchronously send the messages saved in the first replicator to the second cluster, thereby effectively reducing the impact of the first replicator on the consumption behavior of the consumers of the first cluster.

[0055] S105: After determining that the target topic of the second cluster successfully saves the message, an Acknowledge character (ACK) is sent to the first replicator, so that the first replicator updates its own consumption cursor.

[0056] Among them, after the producer in the first replicator successfully sends the message to the second cluster, it is necessary to notify the first replicator that the message is sent successfully to avoid the first replicator wasting time sending the message again and updating its own consumption cursor in time so as to send other messages to the second cluster as soon as possible.

[0057] Based on the above process shown in S101-S105, it can be simply summarized as follows: Figure 1b The scene shown, Figure 1b The specific implementation process of the scenario shown includes:

[0058] 1. After producer A sends the first message to topic A of the first cluster, based on the consumer cursor of the first replicator, the first message is obtained from the message queue indicated by topic A of the first cluster, and the first message is saved in the first replicator;

[0059] 2. When the first cluster pushes the first message to consumer A who subscribes to its own topic A, the producer in the first replicator is called to send the first message saved in the first replicator to topic A of the second cluster, so that topic A of the second cluster saves the first message;

[0060] 3. After determining that the topic A of the second cluster successfully saves the first message, an ACK is sent to the first replicator so that the first replicator updates its own consumption cursor;

[0061] 4. After producer B sends the second message to topic A of the second cluster, based on the consumption cursor of the second replicator, the first message and the second message are obtained from the message queue indicated by topic A of the second cluster, and the first message and the second message are saved in the second replicator;

[0062] 5. Control the second cluster to push the first message and the second message to consumer B who subscribes to its own topic A.

[0063] From the above specific implementation process, it can be seen that based on the first replicator preset in the first cluster, the first cluster does not need to consume machine resources to copy its own messages to the second cluster, thereby ensuring that the performance of the first cluster is not affected. For the second cluster, since the first replicator contains the producer of the second cluster, based on the data transmission characteristics between the producer and the message middleware, it can be seen that the producer in the first replicator can be used to copy the messages in the first cluster to the second cluster, which will also not occupy the machine resources of the second cluster, thereby ensuring that the performance of the second cluster is not affected. In addition, because the first replicator can be built into the broker node process of the first cluster, it can also eliminate the cross-process and cross-network replication losses between data, and increase the speed of copying the messages of the first cluster to the second cluster.

[0064] It should be emphasized that a replicator may also be provided in the second cluster. Based on the replicator in the second cluster, the first cluster and the second cluster may replicate data with each other. For the specific implementation process, please refer to S106-110 below.

[0065] S106: Based on the second replicator being a consumer of the second cluster, a consumption cursor of the second replicator is pre-established.

[0066] The second replicator is preset in the second cluster, and the second replicator includes the producer of the first cluster. The consumption cursor of the second replicator is used to indicate the ID of the message that can be consumed by the second replicator in the message queue shown in the second cluster. Generally speaking, the second replicator can be built into the broker node process of the second cluster.

[0067] Since the second cluster is a message middleware, based on the characteristics of the message middleware, it can be known that as long as the second replicator subscribes to each topic in the second cluster, the second cluster will push the messages that can be consumed by the second replicator in the message queue corresponding to each topic to the second replicator according to the consumption cursor of the second replicator.

[0068] S107: When the second cluster is not faulty, the messages produced by the producers of the second cluster are saved in the target topic of the second cluster; and when the first cluster is faulty, the messages produced by the producers of the first cluster are saved in the target topic of the second cluster.

[0069] If the value of the message ID of the message produced by the producer of the second cluster is not greater than the value of the message ID indicated by the consumer cursor of the second replicator, the message is saved in the target topic of the second cluster.

[0070] Optionally, when the value of the message ID of the message produced by the producer of the second cluster is greater than the value of the message ID indicated by the consumer cursor of the second replicator, it is prohibited to save the message in the second cluster and the message is deleted, thereby ensuring the idempotence of the data.

[0071] It should be noted that the messages produced by the producers of the second cluster and the messages produced by the producers of the first cluster both include data items and metadata of the data items. The metadata of the data items includes the message ID and the cluster identifier of the cluster to which the producer of the data item belongs.

[0072] S108: Based on the consumption cursor of the second replicator, obtain a message from the message queue corresponding to the target topic of the second cluster, and save the message to the second replicator.

[0073] S109: Call the producer in the second replicator to send the message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the message.

[0074] When the message stored in the second replicator does not include the cluster identifier of the first cluster, the producer in the second replicator is called to send the message stored in the second replicator to the first cluster so that the target topic of the first cluster stores the message.

[0075] Optionally, when the message stored in the second replicator includes the cluster identifier of the first cluster, it is prohibited to send the message stored in the second replicator to the first cluster, thereby avoiding sending the message owned by the first cluster to the first cluster in a loop for multiple times.

[0076] S110: After determining that the target topic of the first cluster successfully saves the message, an ACK is sent to the second replicator, so that the second replicator updates its own consumption cursor.

[0077] Among them, based on the producer of the first cluster shown in the second replicator, after successfully sending the message to the first cluster, the first cluster needs to notify the second replicator that the message was sent successfully to avoid the second replicator wasting time sending the message again and updating its own consumption cursor in time to send other messages to the first cluster as soon as possible.

[0078] It should be noted that based on the above process shown in S101-S110, message replication between multiple clusters can be achieved, for example, the message of the Beijing cluster is replicated to the Shanghai cluster and the Shenzhen cluster, the message of the Shanghai cluster is replicated to the Beijing cluster and the Shenzhen cluster, and the message of the Shenzhen cluster is replicated to the Beijing cluster and the Shanghai cluster. Correspondingly, it is also possible to unidirectionally formulate the replication of messages of multiple clusters to the same cluster, for example, the message of the Shenzhen cluster, the message of the Beijing cluster and the message of the Shanghai cluster are replicated to the Xi'an cluster.

[0079] In summary, compared with the prior art, this embodiment is based on the first replicator that is pre-set in the first cluster and includes the producer of the second cluster. It can copy the messages contained in the first cluster to the second cluster without consuming the machine resources of the first cluster, thereby copying the messages in the cluster to other clusters without affecting the performance of the cluster, effectively improving the disaster recovery capability of the cluster. In addition, because the first replicator can be built into the broker node process of the first cluster, it can also eliminate the cross-process and cross-network replication loss between data, and improve the speed of copying the messages of the first cluster to the second cluster.

[0080] It should be noted that S101 mentioned in the above embodiment is an optional implementation of the data processing method based on message middleware described in this application. In addition, S110 mentioned in the above embodiment is also an optional implementation of the data processing method based on message middleware described in this application. To this end, the process shown in the above embodiment can be summarized as follows: Figure 2 The method shown.

[0081] like Figure 2 FIG. 1 is a flow chart of another data processing method based on message middleware provided in an embodiment of the present application, comprising the following steps:

[0082] S201: Save the message produced by the producer of the first cluster to the target topic of the first cluster.

[0083] The message includes a data item, and the target topic is a topic corresponding to the business type to which the data item belongs.

[0084] S202: Based on the consumption cursor of the first replicator, obtain a message from a message queue corresponding to a target topic of the first cluster, and save the message to the first replicator.

[0085] The first replicator is preset in the first cluster, and the first replicator includes the producer of the second cluster; the first cluster and the second cluster belong to different regions respectively; the consumption cursor of the first replicator is pre-established based on the first replicator as a consumer of the first cluster.

[0086] S203: Call the producer in the first replicator to send the message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the message.

[0087] S204: After determining that the target topic of the second cluster successfully saves the message, a confirmation character is sent to the first replicator, so that the first replicator updates its own consumption cursor.

[0088] In summary, compared with the prior art, this embodiment is based on the first replicator that is pre-set in the first cluster and includes the producer of the second cluster. It can copy the messages contained in the first cluster to the second cluster without consuming the machine resources of the first cluster, thereby copying the messages in the cluster to other clusters without affecting the performance of the cluster, effectively improving the disaster recovery capability of the cluster. In addition, because the first replicator can be built into the broker node process of the first cluster, it can also eliminate the cross-process and cross-network replication loss between data, and improve the speed of copying the messages of the first cluster to the second cluster.

[0089] Corresponding to the data processing method based on message middleware provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides a data processing device based on message middleware.

[0090] like Figure 3 FIG. 1 is a schematic diagram of the architecture of a data processing device based on a message middleware provided in an embodiment of the present application, including:

[0091] The message storage unit 100 is used to store the message produced by the producer of the first cluster in the target topic of the first cluster; the message includes a data item; the target topic is a topic corresponding to the business type to which the data item belongs.

[0092] The message storage unit 100 is also used to: in the event of a failure of the first cluster, store the messages produced by the producer of the first cluster in the target topic of the second cluster.

[0093] The message storage unit 100 is further used to: when the second cluster is not faulty, store the message produced by the producer of the second cluster in the target topic of the second cluster.

[0094] In an embodiment of the present application, the message also includes metadata of the data item, and the metadata includes the ID of the message and the cluster identifier of the cluster to which the producer of the data item belongs.

[0095] The message saving unit 100 is specifically used to save the message into the target topic of the second cluster when the value of the ID of the message produced by the producer of the second cluster is not greater than the value of the ID of the message indicated by the consumption cursor of the second replicator.

[0096] The message storage unit 100 is specifically used to: when the value of the ID of the message produced by the producer of the second cluster is greater than the value of the ID of the message shown by the consumer cursor of the second replicator, prohibit the message from being saved in the second cluster and delete the message.

[0097] The message consumption unit 200 is used to obtain messages from the message queue corresponding to the target topic of the first cluster based on the consumption cursor of the first replicator, and save the messages to the first replicator; the first replicator is preset in the first cluster, and the first replicator includes the producer of the second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption cursor of the first replicator is pre-established based on the first replicator as a consumer of the first cluster.

[0098] The message consumption unit 200 is also used to: based on the consumption cursor of the second replicator, obtain messages from the message queue corresponding to the target topic of the second cluster, and save the messages to the second replicator; the second replicator is preset in the second cluster, and the second replicator includes the producer of the first cluster; the consumption cursor of the second replicator is pre-established based on the second replicator as a consumer of the second cluster.

[0099] The message replication unit 300 is used to call the producer in the first replicator to send the message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the message.

[0100] The message replication unit 300 is further used to call the producer in the second replicator to send the message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the message.

[0101] The message replication unit 300 is specifically used for: when the message stored in the second replicator does not contain the cluster identifier of the first cluster, calling the producer in the second replicator to send the message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the message.

[0102] The message duplication unit 300 is specifically configured to prohibit sending the message stored in the second duplicator to the first cluster when the message stored in the second duplicator contains the cluster identifier of the first cluster.

[0103] The cursor updating unit 400 is used to send a confirmation character to the first replicator after determining that the target topic of the second cluster successfully saves the message, so that the first replicator updates its own consumption cursor.

[0104] The cursor updating unit 400 is further configured to: after determining that the target topic of the first cluster successfully saves the message, send a confirmation character to the second replicator, so that the second replicator updates its own consumption cursor.

[0105] In summary, compared with the prior art, this embodiment is based on the first replicator that is pre-set in the first cluster and includes the producer of the second cluster. It can copy the messages contained in the first cluster to the second cluster without consuming the machine resources of the first cluster, thereby copying the messages in the cluster to other clusters without affecting the performance of the cluster, effectively improving the disaster recovery capability of the cluster. In addition, because the first replicator can be built into the broker node process of the first cluster, it can also eliminate the cross-process and cross-network replication loss between data, and improve the speed of copying the messages of the first cluster to the second cluster.

[0106] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the data processing method based on the message middleware provided by the present application.

[0107] The present application also provides a data processing device based on a message middleware, including: a processor, a memory, and a bus. The processor and the memory are connected via a bus, the memory is used to store programs, and the processor is used to run programs, wherein when the program is running, the data processing method based on the message middleware provided by the present application is executed, including the following steps:

[0108] Saving the message produced by the producer of the first cluster to the target topic of the first cluster; the message includes a data item; the target topic is a topic corresponding to the business type to which the data item belongs;

[0109] Based on the consumption cursor of the first replicator, the message is obtained from the message queue corresponding to the target topic of the first cluster, and the message is saved in the first replicator; the first replicator is preset in the first cluster, and the first replicator includes the producer of the second cluster; the first cluster and the second cluster belong to different regions respectively; the consumption cursor of the first replicator is pre-established based on the first replicator as a consumer of the first cluster;

[0110] Calling the producer in the first replicator to send the message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the message;

[0111] After determining that the target topic of the second cluster successfully saves the message, a confirmation character is sent to the first replicator, so that the first replicator updates its own consumption cursor.

[0112] Specifically, based on the above embodiment, it also includes:

[0113] In the event of a failure of the first cluster, messages produced by producers of the first cluster are saved in a target topic of the second cluster.

[0114] Specifically, based on the above embodiment, it also includes:

[0115] When the second cluster is not faulty, save the message produced by the producer of the second cluster to the target topic of the second cluster;

[0116] Based on the consumption cursor of the second replicator, the message is obtained from the message queue corresponding to the target topic of the second cluster, and the message is saved in the second replicator; the second replicator is preset in the second cluster, and the second replicator includes the producer of the first cluster; the consumption cursor of the second replicator is pre-established based on the second replicator as a consumer of the second cluster;

[0117] Calling the producer in the second replicator to send the message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the message;

[0118] After determining that the target topic of the first cluster successfully saves the message, a confirmation character is sent to the second replicator to enable the second replicator to update its own consumption cursor.

[0119] Specifically, based on the above embodiment, the message further includes metadata of the data item, and the metadata includes an ID of the message;

[0120] The step of saving the message produced by the producer of the second cluster to the target topic of the second cluster includes:

[0121] When the value of the ID of the message produced by the producer of the second cluster is not greater than the value of the ID of the message indicated by the consumer cursor of the second replicator, the message is saved in the target topic of the second cluster.

[0122] Specifically, based on the above embodiment, it also includes:

[0123] When the value of the ID of the message produced by the producer of the second cluster is greater than the value of the ID of the message indicated by the consumer cursor of the second replicator, it is prohibited to save the message in the second cluster and the message is deleted.

[0124] Specifically, based on the above embodiment, the message further includes metadata of the data item, and the metadata includes a cluster identifier of a cluster to which a producer of the data item belongs;

[0125] The calling of the producer in the second replicator to send the message stored in the second replicator to the first cluster so that the target topic of the first cluster stores the message includes:

[0126] When the message stored in the second replicator does not include the cluster identifier of the first cluster, the producer in the second replicator is called to send the message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the message.

[0127] Specifically, based on the above embodiment, it also includes:

[0128] In a case where the message stored in the second replicator includes the cluster identifier of the first cluster, sending the message stored in the second replicator to the first cluster is prohibited.

[0129] If the functions described in the method of the embodiment of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the part of the embodiment of the present application that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computing device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories, random access memories, magnetic disks or optical disks.

[0130] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0131] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method based on message middleware, characterized in that: include: Saving the first message produced by the producer of the first cluster to the target topic of the first cluster; The first message includes a data item; the target topic is a topic corresponding to the business type to which the data item belongs; Based on the consumption cursor of the first replicator, the first message is obtained from the message queue corresponding to the target topic of the first cluster, and the first message is saved in the first replicator; the first replicator is preset in the first cluster, and the first replicator includes the producer of the second cluster; the first cluster and the second cluster belong to different regions respectively; the consumption cursor of the first replicator is pre-established based on the first replicator as a consumer of the first cluster; Calling the producer in the first replicator to send the first message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the first message; After determining that the target topic of the second cluster successfully saves the first message, a confirmation character is sent to the first replicator to enable the first replicator to update its own consumption cursor.

2. The method according to claim 1, characterized in that Also includes: In the event of a failure of the first cluster, the first message produced by the producer of the first cluster is saved in the target topic of the second cluster.

3. The method according to claim 1, characterized in that: Also includes: When the second cluster is not faulty, saving the second message produced by the producer of the second cluster to the target topic of the second cluster; Based on the consumption cursor of the second replicator, the second message is obtained from the message queue corresponding to the target topic of the second cluster, and the second message is saved in the second replicator; the second replicator is preset in the second cluster, and the second replicator includes the producer of the first cluster; the consumption cursor of the second replicator is pre-established based on the second replicator as a consumer of the second cluster; Calling the producer in the second replicator to send the second message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the second message; After determining that the target topic of the first cluster successfully saves the second message, a confirmation character is sent to the second replicator to enable the second replicator to update its own consumption cursor.

4. The method according to claim 3, characterized in that The second message also includes metadata of the data item, the metadata including an ID of the second message; The step of saving the second message produced by the producer of the second cluster to the target topic of the second cluster includes: When the value of the ID of the second message produced by the producer of the second cluster is not greater than the value of the ID of the message indicated by the consumer cursor of the second replicator, the second message is saved in the target topic of the second cluster.

5. The method according to claim 4, characterized in that Also includes: When the value of the ID of the second message produced by the producer of the second cluster is greater than the value of the ID of the message indicated by the consumer cursor of the second replicator, it is prohibited to save the second message in the second cluster and the second message is deleted.

6. The method according to claim 3, characterized in that: The second message also includes metadata of the data item, the metadata including a cluster identifier of a cluster to which a producer of the data item belongs; The calling the producer in the second replicator to send the second message stored in the second replicator to the first cluster so that the target topic of the first cluster stores the second message includes: When the second message stored in the second replicator does not include the cluster identifier of the first cluster, the producer in the second replicator is called to send the second message stored in the second replicator to the first cluster, so that the target topic of the first cluster stores the second message.

7. The method according to claim 6, characterized in that Also includes: In a case where the second message stored in the second replicator includes the cluster identifier of the first cluster, sending the second message stored in the second replicator to the first cluster is prohibited.

8. A data processing device based on message middleware, characterized in that: include: A message saving unit, used to save the first message produced by the producer of the first cluster to the target topic of the first cluster; The first message includes a data item; the target topic is a topic corresponding to the business type to which the data item belongs; a message consumption unit, configured to obtain the first message from a message queue corresponding to a target topic of the first cluster based on a consumption cursor of the first replicator, and save the first message to the first replicator; the first replicator is preset in the first cluster, and the first replicator includes a producer of the second cluster; the first cluster and the second cluster belong to different regions respectively; the consumption cursor of the first replicator is pre-established based on the first replicator being a consumer of the first cluster; a message replication unit, configured to call the producer in the first replicator, and send the first message stored in the first replicator to the second cluster, so that the target topic of the second cluster stores the first message; The cursor updating unit is used to send a confirmation character to the first replicator after determining that the target topic of the second cluster successfully saves the first message, so that the first replicator updates its own consumption cursor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the data processing method based on message middleware according to any one of claims 1-7.

10. A data processing device based on message middleware, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program executes the data processing method based on message middleware according to any one of claims 1 to 7 when running.