Message processing method, device, electronic device and storage medium

Abstract

The present invention provides a message processing method, device, electronic device and storage medium, the method is applied to a message routing middle layer, the method includes: receiving a message sent by a client; based on the request header of the message, routing the message to a corresponding topic in a cluster federation; the cluster federation includes multiple clusters, each cluster includes multiple topics. The method, device, electronic device and storage medium provided by the present invention route and forward the message sent by the client to the corresponding topic in the cluster federation through a message routing middle layer, when a large number of messages need to be processed, the multiple topics to which these messages belong can be allocated to multiple clusters in the cluster federation, thereby reducing the load for a single Kafka cluster, solving the problem of performance degradation of a single Kafka cluster when there are too many topics through routing, ensuring the performance of the Kafka cluster, and improving the high concurrent processing efficiency of messages.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a message processing method, device, electronic device and storage medium. Background Art

[0002] Kafka is a high-throughput distributed publish-subscribe messaging system that is widely used in the field of big data processing.

[0003] At present, message processing systems are usually built based on Kafka clusters. However, when the number of topics in a Kafka cluster reaches tens of thousands, the total number of partitions will be large, and disk read and write will degenerate into random read and write, which will seriously affect performance and cannot meet business needs in this scenario. Summary of the invention

[0004] The present invention provides a message processing method, device, electronic device and storage medium, which are used to solve the defect in the prior art that the performance of a Kafka cluster decreases when the number of topics is too large, and to ensure the performance of a Kafka cluster when the number of topics is too large.

[0005] The present invention provides a message processing method, which is applied to a message routing intermediate layer, and comprises:

[0006] Receive messages sent by the client;

[0007] Based on the request header of the message, the message is routed to a corresponding topic in a cluster federation; the cluster federation includes multiple clusters, and each cluster includes multiple topics.

[0008] According to a message processing method provided by the present invention, routing the message to a corresponding topic in a cluster federation based on a request header of the message includes:

[0009] Determine a target cluster corresponding to the message based on a request header of the message;

[0010] Based on the target cluster and the topic to which the message belongs, the message is routed to the topic to which the message belongs in the target cluster.

[0011] According to a message processing method provided by the present invention, the message is routed to the topic to which the message belongs in the target cluster based on the target cluster and the topic to which the message belongs, and the method also includes:

[0012] If the target cluster does not include the topic to which the message belongs, a prompt message is sent to the target cluster so that the target cluster can create the topic to which the message belongs.

[0013] According to a message processing method provided by the present invention, determining a target cluster corresponding to the message based on a request header of the message includes:

[0014] The target cluster is determined based on a request header of the message and the number of clusters included in the cluster federation.

[0015] According to a message processing method provided by the present invention, the determining the target cluster based on the request header of the message and the number of clusters included in the cluster federation includes:

[0016] The target cluster is determined based on a request header of the message, the number of clusters, and the number of topics included in each cluster.

[0017] According to a message processing method provided by the present invention, the request header of the message includes a first key and a second key;

[0018] The routing of the message to a corresponding topic in the cluster federation based on the request header of the message includes:

[0019] Based on the first key, routing the message to a corresponding topic in the cluster federation;

[0020] Based on the second key, the message is routed to a corresponding partition in the corresponding topic.

[0021] A message processing method provided by the present invention further includes:

[0022] Based on the topic subscribed by the client, pull the message in the topic subscribed by the client from the cluster federation;

[0023] Forward the messages in the topic subscribed by the client to the client.

[0024] The present invention also provides a message processing device, which is applied to a message routing intermediate layer, and comprises:

[0025] A receiving module, used to receive messages sent by the client;

[0026] A routing module is used to route the message to a corresponding topic in a cluster federation based on a request header of the message; the cluster federation includes multiple clusters, and each cluster includes multiple topics.

[0027] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the message processing methods described above are implemented.

[0028] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the message processing methods described above are implemented.

[0029] The message processing method, device, electronic device and storage medium provided by the present invention route and forward the messages sent by the client to the corresponding topics in the cluster federation through the message routing intermediate layer. When a large number of messages need to be processed, the multiple topics to which these messages belong can be distributed to multiple clusters in the cluster federation, thereby reducing the load on a single Kafka cluster. The problem of performance degradation of a single Kafka cluster when there are too many topics is solved through routing, the performance of the Kafka cluster is guaranteed, the high-concurrency processing efficiency of messages is improved, and the business needs in high-concurrency scenarios can be met. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 It is a flowchart of the message processing method provided by the present invention;

[0032] Figure 2 It is a structural schematic diagram of the message processing system provided by the present invention;

[0033] Figure 3 It is a structural schematic diagram of a message processing device provided by the present invention;

[0034] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Message queues are important components in distributed systems. They mainly solve problems such as application coupling, asynchronous messages, and traffic clipping, and realize high-performance, high-availability, scalable, and eventually consistent architectures. They are indispensable middleware for large distributed systems. Currently, traditional message queues (MQ) include RabbitMQ, ActiveMQ, and RocketMQ.

[0037] RabbitMQ is a message queue system developed in Erlang language. It is very heavyweight and suitable for enterprise-level development. The core is that producers will not send messages directly to queues. Messages are queued in the central queue before being sent to clients. The disadvantage is that it is not conducive to secondary development and maintenance, runs slowly, and the message is large after encapsulation. ActiveMQ is an open source message queue system. A small amount of code can efficiently implement advanced application scenarios and support pluggable transmission protocols. The disadvantage is that the community activity is not very high. It is not suitable for application scenarios with thousands of queues and there is a possibility of message loss. RocketMQ is written in Java language. Its performance stability and reliability are trustworthy. It adopts the publish / subscribe model, but lacks integration and compatibility with the surrounding ecosystem.

[0038] Kafka is a distributed publish-subscribe messaging system and a powerful queue that can handle large amounts of data and can pass messages from one endpoint to another. Kafka is suitable for both offline and online message consumption. Kafka messages are retained on disk and replicated within the cluster to prevent data loss. Kafka is built on the ZooKeeper synchronization service. It can be very well integrated with Storm and Spark for real-time streaming data analysis. It has reliability, scalability, durability and performance, and is widely used in the field of big data processing. Kafka can solve the following problems: 1) Distributed system indicator collection; 2) Log aggregation; 3) Stream processing; 4) Data generation and consumption security in distributed scenarios.

[0039] In this regard, an embodiment of the present invention provides a message processing method based on Kafka. Figure 1 It is a flow chart of the message processing method provided by the present invention, such as Figure 1 As shown, the execution subject of the method is the message routing middle layer, and the method includes:

[0040] Step 110, receiving a message sent by the client;

[0041] Step 120, based on the request header of the message, routing the message to the corresponding topic in the cluster federation; the cluster federation includes multiple clusters, and each cluster includes multiple topics.

[0042] Specifically, considering that the existing message processing system usually only includes a single Kafka cluster, for scenarios that need to process a large number of messages, such as business scenarios such as orders and logs, when the number of topics in a single Kafka cluster reaches tens of thousands, it will lead to a large number of total partitions, and disk reading and writing will degenerate into random reading and writing, which will seriously affect performance and cannot meet business needs in this scenario. To address this problem, an embodiment of the present invention replaces the existing single Kafka cluster with a cluster federation. The cluster federation here can be composed of multiple Kafka clusters. On this basis, a highly available message routing middle layer is added between the cluster federation and the client. The message routing middle layer can distribute the messages sent by the client to each cluster in the cluster federation according to the topic type, so that multiple related topics in this scenario can be allocated to multiple clusters, and then the pressure of the original single cluster can be shared, solving the problem of performance degradation of a single Kafka cluster when there are too many topics.

[0043] The message routing middle layer can be used as a proxy communication between the client and the cluster federation. After the message routing middle layer receives the message sent by the client, it can perform message routing forwarding according to the request header of the message, and route the message to the corresponding topic in the cluster federation. Here, the request header of the message may include the header information of the message, for example, the key information (key). The message routing middle layer can specifically perform message routing forwarding according to the routing strategy for the message request header. The routing strategy may include, for example, which Kafka cluster the message corresponding to the request header A should be assigned to, which Kafka cluster the message corresponding to the request header B should be assigned to, etc. The routing strategy can be generated by the message routing middle layer based on the client's customized rules, or it can be generated by the message routing middle layer based on its own rules. The embodiment of the present invention does not make specific restrictions on this. In addition, the embodiment of the present invention does not make specific restrictions on the number of clusters included in the cluster federation and the number of topics included in each cluster.

[0044] It is understandable that the various Kafka clusters in the cluster federation do not need to communicate with each other. They only need to communicate with the message routing middle layer and receive messages routed by the message routing middle layer, which can avoid the problem of low timing caused by cross-cluster message routing. In addition, the message routing middle layer will maintain the information in the cluster federation, such as the number of clusters included in the cluster federation, the number of topics included in each cluster, the number of partitions included in each topic, etc., and then formulate routing strategies for message routing and forwarding based on this information. For the client, there is no need to pay attention to how many clusters the topic is assigned to. Even if the number of clusters in the cluster federation changes, it will not affect the client.

[0045] It should be noted that, unlike the existing message processing method, the client directly sends messages to a single Kafka cluster. When there are too many topics in a single Kafka cluster, the single Kafka cluster will change from sequential writing to random writing, and the throughput will be reduced. The embodiment of the present invention applies a message routing middle layer to route and forward the messages sent by the client to the cluster federation, and allocate multiple topics to multiple Kafka clusters in the cluster federation, thereby reducing the load for a single Kafka cluster. The problem of performance degradation of a single Kafka cluster when there are too many topics is solved through routing. Each Kafka cluster can still ensure the sequential reading and writing of messages, thereby improving the message processing efficiency, ensuring the high throughput of the entire system, and being able to meet business needs in high-concurrency scenarios.

[0046] The method provided in the embodiment of the present invention routes and forwards the messages sent by the client to the corresponding topics in the cluster federation through the message routing intermediate layer. When a large number of messages need to be processed, the multiple topics to which these messages belong can be distributed to multiple clusters in the cluster federation, thereby reducing the load on a single Kafka cluster. The problem of performance degradation of a single Kafka cluster when there are too many topics is solved through routing, the performance of the Kafka cluster is guaranteed, the high-concurrency processing efficiency of messages is improved, and the business needs in high-concurrency scenarios can be met.

[0047] Based on any of the above embodiments, in step 120, routing the message to a corresponding topic in the cluster federation based on the request header of the message includes:

[0048] Based on the request header of the message, determine the target cluster corresponding to the message;

[0049] Based on the target cluster and the topic to which the message belongs, the message is routed to the topic to which the message belongs in the target cluster.

[0050] Specifically, the specific method of message routing forwarding can be that the message routing middle layer determines which specific cluster in the cluster federation, that is, the target cluster, to assign the message to based on the request header of the current message and the routing strategy. Then, the message routing middle layer communicates with the target cluster in the cluster federation, and routes the message to the topic to which the message belongs in the target cluster based on the topic to which the message belongs. Here, the specific routing method can be determined based on the routing information pre-configured in the message routing middle layer, and the routing information can be customized according to different business needs, such as hash, key, int value, modulus, etc., which is not specifically limited in the embodiment of the present invention.

[0051] Based on any of the above embodiments, based on the target cluster and the topic to which the message belongs, the message is routed to the topic to which the message belongs in the target cluster, and the method also includes:

[0052] If the target cluster does not include the topic to which the message belongs, a prompt message is sent to the target cluster so that the target cluster can create the topic to which the message belongs.

[0053] Specifically, considering that if the target cluster does not have the topic to which the message belongs, the message routing middle layer will not be able to determine where to route the message after receiving the message sent by the client. In this regard, in an embodiment of the present invention, the message routing middle layer can obtain the topics included in each cluster from the cluster federation. If it is detected that the target cluster does not include the topic to which the message belongs, a prompt message can be sent to the target cluster so that the target cluster can create the topic to which the message belongs based on the prompt message. When the message is subsequently routed, the message can be routed to the topic; if it is detected that the target cluster includes the topic to which the message belongs, no other operation is required, and the message can be directly routed to the topic when the message is subsequently routed.

[0054] Based on any of the foregoing embodiments, determining a target cluster corresponding to the message based on a request header of the message includes:

[0055] The target cluster is determined based on the request header of the message and the number of clusters included in the cluster federation.

[0056] Specifically, before determining the target cluster, the message routing intermediate layer can obtain the number of clusters included in the cluster federation from the cluster federation, thereby updating the routing strategy. Here, the routing strategy can be determined only according to the number of clusters included in the cluster federation. For example, the number of clusters included in the cluster federation is 3, and the routing strategy can be a strategy for allocating these messages to 3 Kafka clusters according to the request headers of different messages. For another example, the number of clusters included in the cluster federation is 5, and the routing strategy can be a strategy for allocating these messages to 5 Kafka clusters according to the request headers of different messages. The routing strategy can also be determined based on the number of clusters included in the cluster federation and other information. For example, the cluster federation includes 3 Kafka clusters, and it is detected that the message write rate of one of the clusters is too slow. The routing strategy can be changed to a strategy for allocating subsequent messages to the remaining two Kafka clusters according to the request headers of different messages. For another example, the cluster federation includes 3 Kafka clusters, and it is detected that the number of topics included in one of the clusters is too large. The routing strategy can be changed to a strategy for allocating subsequent messages to the remaining two Kafka clusters according to the request headers of different messages. The embodiments of the present invention do not specifically limit this.

[0057] Then, the message routing middle layer can determine which specific cluster in the cluster federation, that is, the target cluster, to assign the message to based on the request header of the current message and the routing policy.

[0058] Based on any of the foregoing embodiments, determining a target cluster based on a request header of a message and the number of clusters included in the cluster federation includes:

[0059] Determine the target cluster based on the message request header, the number of clusters, and the number of topics included in each cluster.

[0060] Specifically, if the routing strategy is determined only based on the number of clusters, there may be a situation where the number of topics included in each cluster in the cluster federation is unevenly distributed, resulting in uneven loads on each cluster. To address this issue, in an embodiment of the present invention, before determining the target cluster, the message routing intermediate layer can obtain the number of clusters included in the cluster federation from the cluster federation, and can also obtain the number of topics included in each cluster in the cluster federation from the cluster federation. At this time, the routing strategy can include that if it is detected that the number of topics included in a certain Kafka cluster is low, the message can be preferentially routed to the cluster, that is, the cluster is used as the target cluster, and if it is detected that the number of topics included in each Kafka cluster is relatively uniform, it can still be determined based on the request header of the message and the number of clusters included in the cluster federation to which the message is assigned, that is, the target cluster.

[0061] Based on any of the above embodiments, the request header of the message includes a first key and a second key;

[0062] Based on the request header of the message, the message is routed to the corresponding topic in the cluster federation, including:

[0063] Based on the first key, route the message to the corresponding topic in the cluster federation;

[0064] Based on the second key, the message is routed to the corresponding partition in the corresponding topic.

[0065] Specifically, the request header of the message may include a first key, which can be used to indicate to which cluster in the cluster federation the message needs to be assigned, and on this basis, the message routing layer can route the message to the corresponding topic in the cluster according to the first key and the known topic to which the message belongs. Considering that each topic is divided into one or more partitions, each partition is composed of a series of ordered, immutable messages and is an ordered queue, therefore, the request header of the message may also include a second key, which can be used to indicate to which partition in the corresponding topic the message needs to be assigned, and according to the second key, the message routing layer can route the message to the partition in the corresponding topic.

[0066] Here, the first key may be added to the request header of the message by the message routing middle layer based on the default rule, the second key may be added to the request header of the message by the client and then forwarded to the message routing middle layer, or may be added to the request header of the message by the message routing middle layer based on the default rule, and the embodiment of the present invention does not specifically limit this. Further, considering that the Kafka cluster is composed of multiple Kafka instances, each instance (server) becomes a broker, in this regard, the request header of the message may also include a third key, and the third key may be used to indicate which broker in the Kafka cluster the message needs to be assigned to.

[0067] Based on any of the above embodiments, the method further includes:

[0068] Based on the topic subscribed by the client, pull the messages in the topic subscribed by the client from the cluster federation;

[0069] Forwards messages in the topics subscribed by the client to the client.

[0070] Specifically, when a client needs to consume a message, the client can be called a consumer. The consumer can subscribe to various topics in the cluster federation. When a consumer subscribes to a topic, the message routing middle layer can pull the message in the topic from the cluster federation according to the topic, and then forward the message in the topic to the consumer for the consumer to consume the message in the topic. Furthermore, the consumer can send a message request to the message routing middle layer, and the message request can carry information used to represent the topic subscribed by the consumer. After the message routing middle layer receives the message request, it can obtain the topic subscribed by the consumer.

[0071] Based on any of the above embodiments, the embodiment of the present invention provides a message processing system based on the Kafka cluster. Figure 2 It is a structural diagram of the message processing system provided by the present invention. Figure 2 As shown, the system includes a client (i.e. Figure 2 Client in), message routing middle layer (i.e. Figure 2 The Middle Layer in the cluster federation (i.e. Figure 2 KafkaCluster Federation) and Zookeeper cluster (i.e. Figure 2 in the Zookeeper Cluster).

[0072] The Zookeeper cluster is used for services such as message delivery in a distributed architecture. Each topic is divided into one or more partitions. Each message in a partition has a continuous sequence number called an offset. The client maintains which offset to consume and needs to provide it to the Zookeeper cluster. In addition, the meta information of each Kafka cluster in the cluster federation needs to be registered and saved in the Zookeeper cluster in the register mode. Here, the meta information may include information such as which server the Kafka cluster creates the topic on, how many partitions the topic has, and which message the client consumes in the partition. The embodiments of the present invention do not specifically limit this.

[0073] When a client needs to send a message, the client can be called a producer. The producer sends the message to the message routing middle layer, which then routes the message to a specific topic in the cluster federation. The message routing middle layer can include a topic cache (i.e. Figure 2 The Topic Cache in the cluster cache pulls cluster information from the cluster federation and topic information in each Kafka cluster (i.e. Figure 2 The cluster information here can be, for example, the number of clusters included in the cluster federation, and the topic information can be, for example, the number of topics, the names of topics, and the number of partitions included in each topic. The obtained relevant information is stored, and when forwarding messages, the routing strategy can be determined based on the relevant information, and the request header (i.e. Figure 2 Header: key) in the message routing assignment.

[0074] When a client needs to consume a message, the client can be called a consumer. The consumer can subscribe to a topic. The message routing middle layer pulls the message in the topic from the cluster federation in a polling manner and forwards the message to the consumer for consumption. The topic cache of the message routing middle layer can also store information such as the offset of consumption, so that the message routing middle layer can pull messages according to the offset.

[0075] Furthermore, the message processing system can also add a monitoring mechanism, which can collect information about each Kafka cluster in the cluster federation and information in the Zookeeper cluster, and monitor each Kafka cluster in the cluster federation based on this information. The Kafka cluster in the cluster federation supports dynamic expansion. When the monitoring mechanism detects that a certain Kafka cluster has problems such as message consumption failure, message consumption rate or write rate being too slow, the Kafka cluster can be expanded, that is, a new Kafka cluster can be added on the basis of the Kafka cluster. For example, if the monitoring mechanism detects that Kafka cluster A consumes 100,000 data per second, it can be expanded, and Kafka cluster B can be added on the basis of Kafka cluster A.

[0076] If the number of clusters in the cluster federation changes, the message routing middle layer will also update the stored relevant information. After receiving a message on a related topic from the message routing middle layer, the message can be routed to the corresponding topic in the newly added Kafka cluster. For example, in the above example, if the message routing middle layer subsequently receives messages corresponding to the same topic in Kafka cluster A, these messages can be routed to the newly added Kafka cluster B.

[0077] Of course, if the monitoring mechanism detects that all Kafka clusters in the cluster federation have problems such as the inability to consume messages or the slow message consumption rate or writing rate, the original cluster federation can also be expanded, that is, a new Kafka cluster can be added on the basis of the original Kafka cluster. For example, the original cluster federation includes 3 Kafka clusters, and the monitoring mechanism detects that the message writing rate of these 3 Kafka clusters is too slow. In this case, the 3 Kafka clusters can be expanded, and 2 new Kafka clusters can be added, so that the number of clusters included in the new cluster federation is changed to 5.

[0078] In addition, a watch mechanism can also be configured in the Zookeeper cluster. When there are event changes, such as the addition, deletion, and modification of topics in the Kafka cluster, the watch mechanism will push them to the message routing middle layer in the form of event messages. The message routing layer can update information based on the event messages, thereby ensuring that the relevant information stored in the message routing layer is the latest, thereby ensuring the smooth progress of subsequent message routing.

[0079] In addition to storing cluster information, topic information on each cluster, and other related information in the cluster federation, the topic cache area of ​​the message routing middle layer will also pre-configure routing information. The routing information here can be customized according to different business needs, such as hash, key, int value, modulus, etc., which is not specifically limited in the embodiment of the present invention. The message routing middle layer can configure multiple topics to multiple Kafka clusters in the cluster federation through the relevant information and routing information in the cluster federation.

[0080] It should be noted that the message routing middle layer only forwards messages based on the message request header and does not perform business processing. The message routing middle layer can implement the communication protocol of the Kafka cluster. Therefore, it can be ensured that the application accesses the message routing middle layer in the same way as the Kafka clusters in the cluster federation. In addition, the message routing middle layer will not affect the order of messages. After the messages are forwarded to the cluster federation, the original order will be maintained. In order to avoid a single point of failure in the message routing middle layer, which will cause the subsequent cluster federation to be unusable, the message routing middle layer can set up multiple components that provide the same function to receive messages sent by the client.

[0081] The system provided by the embodiment of the present invention routes and forwards the messages sent by the client to the corresponding topics in the cluster federation through the message routing intermediate layer. When a large number of messages need to be processed, the multiple topics to which these messages belong can be distributed to multiple clusters in the cluster federation, thereby reducing the load on a single Kafka cluster. The problem of performance degradation of a single Kafka cluster when there are too many topics is solved through routing, the performance of the Kafka cluster is guaranteed, the high-concurrency processing efficiency of messages is improved, and the business needs in high-concurrency scenarios can be met.

[0082] The message processing device provided by the present invention is described below. The message processing device described below and the message processing method described above can be referenced to each other.

[0083] Based on any of the above embodiments, an embodiment of the present invention provides a message processing device. Figure 3 is a schematic diagram of the structure of the message processing device provided by the present invention, such as Figure 3 As shown, the execution subject of the device is a message routing intermediate layer, and the device includes:

[0084] The receiving module 310 is used to receive a message sent by a client;

[0085] The routing module 320 is used to route the message to the corresponding topic in the cluster federation based on the request header of the message; the cluster federation includes multiple clusters, and each cluster includes multiple topics.

[0086] Specifically, considering that the existing message processing system usually only includes a single Kafka cluster, for scenarios that need to process a large number of messages, such as business scenarios such as orders and logs, when the number of topics in a single Kafka cluster reaches tens of thousands, it will lead to a large number of total partitions, and disk reading and writing will degenerate into random reading and writing, which will seriously affect performance and cannot meet business needs in this scenario. To address this problem, an embodiment of the present invention replaces the existing single Kafka cluster with a cluster federation. The cluster federation here can be composed of multiple Kafka clusters. On this basis, a highly available message routing middle layer is added between the cluster federation and the client. The message routing middle layer can distribute the messages sent by the client to each cluster in the cluster federation according to the topic type, so that multiple related topics in this scenario can be allocated to multiple clusters, and then the pressure of the original single cluster can be shared, solving the problem of performance degradation of a single Kafka cluster when there are too many topics.

[0087] The message routing middle layer can be used as a proxy communication between the client and the cluster federation. After the message routing middle layer receives the message sent by the client, it can perform message routing forwarding according to the request header of the message, and route the message to the corresponding topic in the cluster federation. Here, the request header of the message may include the header information of the message, for example, the key information (key). The message routing middle layer can specifically perform message routing forwarding according to the routing strategy for the message request header. The routing strategy may include, for example, which Kafka cluster the message corresponding to the request header A should be assigned to, which Kafka cluster the message corresponding to the request header B should be assigned to, etc. The routing strategy can be generated by the message routing middle layer based on the client's customized rules, or it can be generated by the message routing middle layer based on its own rules. The embodiment of the present invention does not make specific restrictions on this. In addition, the embodiment of the present invention does not make specific restrictions on the number of clusters included in the cluster federation and the number of topics included in each cluster.

[0088] It is understandable that the various Kafka clusters in the cluster federation do not need to communicate with each other. They only need to communicate with the message routing middle layer and receive messages routed by the message routing middle layer, which can avoid the problem of low timing caused by cross-cluster message routing. In addition, the message routing middle layer will maintain the information in the cluster federation, such as the number of clusters included in the cluster federation, the number of topics included in each cluster, the number of partitions included in each topic, etc., and then formulate routing strategies for message routing and forwarding based on this information. For the client, there is no need to pay attention to how many clusters the topic is assigned to. Even if the number of clusters in the cluster federation changes, it will not affect the client.

[0089] It should be noted that, unlike the existing message processing method, the client directly sends messages to a single Kafka cluster. When there are too many topics in a single Kafka cluster, the single Kafka cluster will change from sequential writing to random writing, and the throughput will be reduced. The embodiment of the present invention applies a message routing middle layer to route and forward the messages sent by the client to the cluster federation, and allocate multiple topics to multiple Kafka clusters in the cluster federation, thereby reducing the load for a single Kafka cluster. The problem of performance degradation of a single Kafka cluster when there are too many topics is solved through routing. Each Kafka cluster can still ensure the sequential reading and writing of messages, thereby improving the message processing efficiency, ensuring the high throughput of the entire system, and being able to meet business needs in high-concurrency scenarios.

[0090] The device provided in the embodiment of the present invention routes and forwards the messages sent by the client to the corresponding topics in the cluster federation through the message routing intermediate layer. When a large number of messages need to be processed, the multiple topics to which these messages belong can be distributed to multiple clusters in the cluster federation, thereby reducing the load on a single Kafka cluster. The problem of performance degradation of a single Kafka cluster when there are too many topics is solved through routing, the performance of the Kafka cluster is guaranteed, the high-concurrency processing efficiency of messages is improved, and the business needs in high-concurrency scenarios can be met.

[0091] Based on any of the above embodiments, the routing module 320 includes:

[0092] The cluster determination submodule is used to determine the target cluster corresponding to the message based on the request header of the message;

[0093] The routing submodule is used to route the message to the topic of the message in the target cluster based on the target cluster and the topic to which the message belongs.

[0094] Based on any of the above embodiments, the device further includes a prompt module, which is used to:

[0095] If the target cluster does not include the topic to which the message belongs, a prompt message is sent to the target cluster so that the target cluster can create the topic to which the message belongs.

[0096] Based on any of the foregoing embodiments, the cluster determination submodule includes a cluster determination unit, configured to:

[0097] The target cluster is determined based on the request header of the message and the number of clusters included in the cluster federation.

[0098] Based on any of the foregoing embodiments, the cluster determination unit is further configured to:

[0099] Determine the target cluster based on the message request header, the number of clusters, and the number of topics included in each cluster.

[0100] Based on any of the above embodiments, the request header of the message includes a first key and a second key;

[0101] The routing module 320 includes:

[0102] A topic routing submodule, used to route the message to a corresponding topic in the cluster federation based on the first key;

[0103] The partition routing submodule is used to route the message to the corresponding partition in the corresponding topic based on the second key.

[0104] Based on any of the above embodiments, the device further includes a pulling module, which is used to:

[0105] Based on the topic subscribed by the client, pull the messages in the topic subscribed by the client from the cluster federation;

[0106] Forwards messages in the topics subscribed by the client to the client.

[0107] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the message processing method, which is applied to the message routing middle layer, and the method includes: receiving a message sent by a client; routing the message to a corresponding topic in a cluster federation based on a request header of the message; the cluster federation includes multiple clusters, and each cluster includes multiple topics.

[0108] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the message processing method provided by the above methods. The method is applied to the message routing middle layer, and the method includes: receiving a message sent by a client; based on the request header of the message, routing the message to the corresponding topic in the cluster federation; the cluster federation includes multiple clusters, and each cluster includes multiple topics.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the message processing method provided by the above-mentioned methods. The method is applied to a message routing middle layer, and the method includes: receiving a message sent by a client; based on a request header of the message, routing the message to a corresponding topic in a cluster federation; the cluster federation includes multiple clusters, and each cluster includes multiple topics.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A message processing method, characterized in that: The method is applied to a message routing intermediate layer, and the method comprises: Receive messages sent by the client; Based on the request header of the message, routing the message to a corresponding topic in a cluster federation; the cluster federation includes a plurality of clusters, each cluster includes a plurality of topics; The routing of the message to a corresponding topic in the cluster federation based on the request header of the message includes: updating a routing policy based on the number of clusters included in the cluster federation, and determining a target cluster based on a request header of the message, the routing policy, and the number of topics included in each cluster; Based on the target cluster and the topic to which the message belongs, routing the message to the topic to which the message belongs in the target cluster; Among them, the routing strategy includes: when it is detected that the message writing rate of one of the clusters is too slow, the subsequent messages are distributed to the remaining clusters according to the request headers of different messages; or, when it is detected that the number of topics included in one of the clusters is too large, the subsequent messages are distributed to the remaining clusters according to the request headers of different messages; the request header of the message includes: a first key and a second key; the first key is used to distribute the message to the corresponding topic of the corresponding cluster; the second key is used to route the message to the corresponding partition of the corresponding topic; the first key is added to the request header of the message by the message routing middle layer based on the default rule; the second key is added to the request header of the message by the client, and then forwarded to the message routing middle layer.

2. The message processing method according to claim 1, characterized in that: The method of routing the message to the topic to which the message belongs in the target cluster based on the target cluster and the topic to which the message belongs also includes: If the target cluster does not include the topic to which the message belongs, a prompt message is sent to the target cluster so that the target cluster can create the topic to which the message belongs.

3. The message processing method according to claim 1 or 2, characterized in that: Also includes: Based on the topic subscribed by the client, pull the message in the topic subscribed by the client from the cluster federation; Forward the messages in the topic subscribed by the client to the client.

4. A message processing device, characterized in that: The device is applied to a message routing intermediate layer, and the device comprises: A receiving module, used to receive messages sent by the client; A routing module, used for routing the message to a corresponding topic in a cluster federation based on a request header of the message; the cluster federation includes a plurality of clusters, each cluster includes a plurality of topics; The routing module is specifically used to update the routing strategy based on the number of clusters included in the cluster federation, and determine the target cluster based on the request header of the message, the routing strategy, and the number of topics included in each cluster; The routing module is further configured to route the message to the topic to which the message belongs in the target cluster based on the target cluster and the topic to which the message belongs; Among them, the routing strategy includes: when it is detected that the message writing rate of one of the clusters is too slow, the subsequent messages are distributed to the remaining clusters according to the request headers of different messages; or, when it is detected that the number of topics included in one of the clusters is too large, the subsequent messages are distributed to the remaining clusters according to the request headers of different messages; the request header of the message includes: a first key and a second key; the first key is used to distribute the message to the corresponding topic of the corresponding cluster; the second key is used to route the message to the corresponding partition of the corresponding topic; the first key is added to the request header of the message by the message routing middle layer based on the default rule; the second key is added to the request header of the message by the client, and then forwarded to the message routing middle layer.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the message processing method according to any one of claims 1 to 3 are implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the message processing method according to any one of claims 1 to 3 are implemented.

Images