Theme message processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a theme message processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining the total message amount of each first preset message theme currently processed by a first service cluster, determining at least one sub-message theme and a sub-service cluster corresponding to each sub-message theme according to the first message total quantity under the condition that the first message total quantity of the first target message theme in the first preset message theme is greater than a preset quantity threshold value; changing a message processing mode in consumption configuration information corresponding to the first target message theme into forwarding processing; and changing a processing execution object in the first cluster configuration information corresponding to each sub-message theme into a corresponding sub-service cluster, so that the sub-service cluster creates a first consumer instance pointing to the corresponding sub-message theme after monitoring that the first cluster configuration information is changed. According to the scheme, the message processing efficiency is improved, and the consumer instance can be flexibly created.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a topic message processing method, apparatus, device, storage medium, and program product. Background Art

[0002] With the rapid development of Internet technology, message queue systems are increasingly used in various businesses and are an important means to solve the message delivery problem of distributed systems. Among them, the message queue system can be set with producers and message topics. The producer is responsible for sending messages to one or more message topics. With the development and iteration of the business, the number of message topics deployed by a single business continues to increase, and the total number of messages in each message topic also continues to increase.

[0003] In the related art, the message topics that the service cluster corresponds to are fixedly configured during the business encoding stage. For message topics with excessive message load, the message processing speed will be slow, resulting in message accumulation, affecting the normal operation of the business, and cannot adapt to real-time large-scale message processing scenarios. Summary of the invention

[0004] The embodiments of the present application provide a topic message processing method, apparatus, equipment, storage medium and program product, which solve the problem that the related technology will result in slow message processing speed for message topics with too heavy message load, causing message accumulation, affecting the normal operation of business, and being unable to adapt to real-time large-scale message processing scenarios. It achieves a more balanced load distribution, avoids performance bottlenecks caused by overload of a single service cluster, improves the message processing speed in real-time large-scale message processing scenarios, and can flexibly create consumer instances to quickly adapt to changing business needs, effectively improving the efficiency and dynamic response capabilities of message processing.

[0005] In a first aspect, an embodiment of the present application provides a topic message processing method, the method comprising:

[0006] Obtain the total number of messages of each first preset message topic currently processed by the first service cluster, and when the total number of first messages of the first target message topic in the first preset message topic is greater than a preset number threshold, determine at least one sub-message topic and a sub-service cluster corresponding to each of the sub-message topics according to the first total number of messages;

[0007] Change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage;

[0008] The processing execution object in the first cluster configuration information corresponding to each of the sub-message topics is changed to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0009] In a second aspect, an embodiment of the present application further provides a topic message processing device, the device comprising:

[0010] a topic cluster determination module, configured to obtain the total amount of messages of each first preset message topic currently processed by the first service cluster, and determine at least one sub-message topic and a sub-service cluster corresponding to each of the sub-message topics according to the first total amount of messages when the total amount of first messages of the first target message topic in the first preset message topic is greater than a preset quantity threshold;

[0011] A first configuration information adjustment module is configured to change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, the message of the first target message topic is forwarded to the corresponding sub-message topic for storage;

[0012] The second configuration information adjustment module is configured to change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0013] In a third aspect, an embodiment of the present application further provides a topic message processing device, the device comprising:

[0014] one or more processors;

[0015] a storage device configured to store one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the topic message processing method described in the embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are configured to execute the topic message processing method described in the embodiment of the present application.

[0018] In the fifth aspect, an embodiment of the present application also provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium, and at least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device executes the topic message processing method described in the embodiment of the present application.

[0019] In an embodiment of the present application, by obtaining the total number of messages of each first preset message topic currently processed by the first service cluster, when the total number of first messages of the first target message topic in the first preset message topic is greater than the preset number threshold, at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic are determined according to the total number of first messages; the message processing mode in the consumption configuration information corresponding to the first target message topic is changed to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the messages of the first target message topic to the corresponding sub-message topic for storage; the processing execution object in the first cluster configuration information corresponding to each sub-message topic is changed to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change of the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the messages in the sub-message topic based on the first consumer instance. In the above scheme, by comparing the total amount of messages of each first preset message topic with the preset quantity threshold, the message load of each preset message topic can be effectively judged, and by determining at least one sub-message topic and the sub-service cluster corresponding to each sub-message topic according to the total amount of the first messages, and adjusting the consumption configuration information corresponding to the first target message topic and the current cluster name in the first cluster configuration information corresponding to each sub-message topic in turn, the pending messages of the high-load message topic can be effectively configured to be forwarded to multiple sub-message topics, and the corresponding multiple sub-service clusters can be configured for parallel processing, so as to achieve a more balanced load distribution, avoid the performance bottleneck caused by overload of a single service cluster, and improve the message processing speed in real-time large-scale message processing scenarios, and also can flexibly create consumer instances to quickly adapt to changing business needs, effectively improving the efficiency and dynamic response capabilities of message processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a topic message processing method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a process for forwarding and fragmenting a message of a first target message topic provided in an embodiment of the present application;

[0022] Figure 3A flowchart of a topic message processing method including a process of configuring a second service cluster to process a second target message topic provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a process for splitting a plurality of first preset message topics into different service clusters for message processing provided in an embodiment of the present application;

[0024] Figure 5 A flowchart of a topic message processing method including a process of configuring a first service cluster to process a message on a second preset message topic provided in an embodiment of the present application;

[0025] Figure 6 A flowchart of a topic message processing method including a process of determining a sub-message topic and a corresponding sub-service cluster provided in an embodiment of the present application;

[0026] Figure 7 A flowchart of a topic message processing method including a process of determining a second target message topic and a corresponding second service cluster provided in an embodiment of the present application;

[0027] Figure 8 A structural block diagram of a topic message processing device provided in an embodiment of the present application;

[0028] Fig. 9 A structural diagram of a topic message processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] The subject message processing method provided in the embodiment of the present application can configure the messages of the first target message topic with a high total message volume to be forwarded to other sub-message topics, and configure a corresponding sub-service cluster for each sub-message topic, so that the messages of the first target message topic with a high total message volume can be diverted to other sub-message topics, and each sub-message topic can be processed by the corresponding sub-service cluster, so as to realize the message fragmentation processing of the message topic with a high total message volume, and achieve the purpose of horizontal capacity expansion. Relevant application scenarios may include: live video, live voice, short video, etc. The several application scenarios listed above are only exemplary and explanatory. In actual applications, the subject message processing method can also be used in the subject message processing in other scenarios, and the embodiment of the present application does not limit this. The subject message processing method, device, equipment, storage medium and program product provided in the embodiment of the present application are intended to solve the problem that the related technology has a slow message processing speed for message topics with excessive message load, resulting in message accumulation, affecting the normal operation of the business, and cannot adapt to real-time large-scale message processing scenarios.

[0032] In the subject message processing method provided in the embodiment of the present application, the executor of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices, and can also be servers and other devices, which are not limited in the embodiment of the present application.

[0033] Figure 1 A flowchart of a topic message processing method provided for an embodiment of the present application. In the message queue system, the producer is a component or application responsible for creating and sending messages, and can interact with the application program interface provided by the message queue service to send messages to the message topic. The message topic can be used for message classification and subscription, and consumers can receive and process messages by subscribing to the message topic of interest, or actively pull messages from the message topic. The specific implementation method can be that the message topic is deployed on a storage server, and the service cluster subscribes to the message topic based on the created consumer instance. The corresponding storage server can send the message in the message topic to the corresponding service cluster, and the service cluster receives and processes the message based on the created consumer instance, or the service cluster actively pulls messages from the message topic based on the created consumer instance. Figure 1 As shown, the topic message processing method includes the following steps:

[0034] Step S101: obtain the total number of messages of each first preset message topic currently processed by the first service cluster. When the total number of first messages of the first target message topic in the first preset message topic is greater than the preset number threshold, determine at least one sub-message topic and the sub-service cluster corresponding to each sub-message topic based on the total number of first messages.

[0035] Among them, the first service cluster is a collection of multiple processing server nodes, which is used to process messages of multiple pre-configured first preset message topics. The total amount of messages can be used to characterize the traffic load of the first preset message topic. Taking the live broadcast scene as an example, the total amount of messages can be the total amount of messages entering the live broadcast room. By comparing the total amount of messages of each first preset message topic with the preset number threshold, it can be determined whether each first preset message topic is in a high-load state. The preset number threshold can be adaptively set by the developer according to the cluster processing capacity and real-time traffic characteristics of the actual application scenario, and this application is not limited here. If the first preset message topic is in a high-load state, the processing speed of the message may be limited due to the limited processing capacity of the first service cluster. If the total amount of first messages of the first target message topic in multiple first preset message topics is greater than the preset number threshold, it can be regarded as a message topic with a high traffic load in the first preset message topic. Due to the limited processing capacity of the first service cluster and the limited number of partitions of the message topic, it is impossible to support parallel processing of messages in high-concurrency scenarios. Therefore, it is possible to consider determining at least one additional sub-message topic based on the total amount of the first message, which is used to segment the messages of the first target message topic into other sub-message topics, and other service clusters can process the messages of the first target message topic in parallel, so as to improve the overall throughput of the system. In one embodiment, different message quantity intervals can be pre-divided, and the first required number of sub-message topics can be determined according to the message quantity interval in which the total amount of the first message is located, and the sub-message topics that meet the first required number are determined from the candidate sub-message topics. In one embodiment, the second number of sub-message topics corresponding to the first preset message topic whose total amount of each message exceeds the preset quantity threshold can be fixedly set, and the sub-message topics that meet the second required number are determined from the candidate sub-message topics. In addition, the sub-service cluster corresponding to each sub-message topic can be determined from the candidate sub-service clusters, without the need for the first service cluster to continue processing the messages of the first target message topic.

[0036] Step S102: Change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage.

[0037] Among them, the consumption configuration message can be used to configure the message processing method of the first service cluster for the first target message topic, and the message processing method can be directly processed or forwarded by the first service cluster. After the message processing method in the consumption configuration information corresponding to the first target message topic is changed to forward processing, the consumption configuration message is synchronized to the first service cluster. During operation, the first service cluster can monitor the consumption configuration message, determine whether the content of the consumption configuration message has changed, and reconfigure the message processing method to adapt to the content change. In this embodiment, since the message processing method in the consumption configuration information corresponding to the first target message topic is changed to forward processing, after the first service cluster monitors the information change of the consumption configuration information, it can forward the message of the first target message topic to the corresponding sub-message topic for storage, without processing the message of the first target message topic.

[0038] In one embodiment, the message processing mode in the consumption configuration information corresponding to the first target message topic is changed to forwarding processing. The specific implementation method may be: set an enable forwarding flag in the consumption configuration information corresponding to the first target message topic, and add preset forwarding rules and the topic name of each sub-message topic to the consumption configuration information. Among them, the enable forwarding flag may indicate that the message of the first target message topic needs to be forwarded, and the preset forwarding rule may be a strategy indicating that the message is routed to different sub-message topics, such as polling routing or hash routing, to achieve balanced distribution of messages to different sub-message topics. By adding the topic name of the sub-message topic, the specific target object of the message forwarding can be determined.

[0039] In one embodiment, a variable "whether to forward" may be set in the message configuration information, and the message processing mode in the consumption configuration information corresponding to the first target message topic may be changed to forward processing by setting the variable "whether to forward" to a true value. In addition, a message forwarding path containing fixed forwarding rules may be pre-set in the message configuration information, and by filling the topic name of the sub-message topic into the message forwarding path, it is possible to specify which sub-message topics the messages of the first target message topic are forwarded to for storage.

[0040] Step S103: Change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0041] Among them, the first cluster configuration information can be used to configure the processing execution object of each sub-message topic. Since the processing execution object in the first cluster configuration information corresponding to the sub-message topic may be empty or other service clusters, it is necessary to change the processing execution object to the corresponding sub-service cluster. In one embodiment, the processing execution object currently set in the first cluster configuration information is other service clusters. Therefore, the processing execution object in the first cluster configuration information corresponding to each sub-message topic is changed to the corresponding sub-service cluster. The specific implementation method can be: modify the current cluster name set in the first cluster configuration information corresponding to each sub-message topic to the cluster name of the corresponding sub-service cluster. In one embodiment, the processing execution object currently set in the first cluster configuration information is empty, and the cluster name of the corresponding sub-service cluster can be added to the first cluster configuration information to complete the setting of the sub-service cluster corresponding to the sub-message topic. During operation, each sub-service cluster can monitor the first cluster configuration information corresponding to the existing sub-consumption topic to determine whether the content of the first cluster configuration information has changed. Among them, the sub-service cluster can create a first consumer instance pointing to a sub-consumption topic by determining that the cluster name in the first cluster configuration information corresponding to a sub-consumption topic is changed to its own cluster name, and process the message in the sub-message topic based on the first consumer instance, thereby realizing the configurable management of consumer instances and flexibly creating and changing consumer instances.

[0042] In one embodiment, Figure 2 A schematic diagram of a process for forwarding and fragmenting a message of a first target message topic provided by an embodiment of the present application, such as Figure 2 As shown, the message of the first target message topic 101 is forwarded to the sub-message topic 103 by the first service cluster 102 according to different preset forwarding rules, and each sub-message topic 103 is correspondingly configured with a sub-service cluster 104 for message processing.

[0043] In the above, by obtaining the total number of messages of each first preset message topic currently processed by the first service cluster, when the total number of first messages of the first target message topic in the first preset message topic is greater than the preset number threshold, at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic are determined according to the total number of first messages; the message processing mode in the consumption configuration information corresponding to the first target message topic is changed to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the messages of the first target message topic to the corresponding sub-message topic for storage; the processing execution object in the first cluster configuration information corresponding to each sub-message topic is changed to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change of the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the messages in the sub-message topic based on the first consumer instance. In the above scheme, by comparing the total amount of messages of each first preset message topic with the preset quantity threshold, the message load of each preset message topic can be effectively judged, and by determining at least one sub-message topic and the sub-service cluster corresponding to each sub-message topic according to the total amount of the first messages, and adjusting the consumption configuration information corresponding to the first target message topic and the current cluster name in the first cluster configuration information corresponding to each sub-message topic in turn, the pending messages of the high-load message topic can be effectively configured to be forwarded to multiple sub-message topics, and the corresponding multiple sub-service clusters can be configured for parallel processing, so as to achieve a more balanced load distribution, avoid the performance bottleneck caused by overload of a single service cluster, and improve the message processing speed in real-time large-scale message processing scenarios, and also can flexibly create consumer instances to quickly adapt to changing business needs, effectively improving the efficiency and dynamic response capabilities of message processing.

[0044] Figure 3 A flowchart of a topic message processing method including a process of configuring a second service cluster to process a second target message topic is provided in an embodiment of the present application. Figure 3 As shown, the topic message processing method includes the following steps:

[0045] Step S201: Obtain the total number of messages of each first preset message topic currently processed by the first service cluster.

[0046] Step S202: When the total amount of first messages of the first target message topic in the first preset message topic is greater than a preset quantity threshold, at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic are determined according to the total amount of first messages.

[0047] Step S203: Change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage.

[0048] Step S204: Change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0049] Step S205: When the total amount of messages in each first preset message topic is less than or equal to a preset quantity threshold, obtain the machine resource utilization of the first service cluster.

[0050] Among them, if the total number of messages of each first preset message topic is less than or equal to the preset number threshold, it can be regarded that the message load of each preset message topic is within an acceptable range, and there is no need to perform message fragmentation processing. However, since the first service cluster processes multiple first preset message topics, message overload problems may occur, resulting in performance bottlenecks in message processing. Therefore, the machine resource utilization of the first service cluster can be obtained, and the machine resource utilization can be used to characterize the degree of utilization of the hardware resources of the first service cluster. For example, the machine resource utilization can be processor utilization, memory utilization, etc. If the machine resource utilization is high, it can be regarded as the first service cluster is in a busy state, and a performance bottleneck may occur, for example, there are many messages waiting to be processed. If the machine resource utilization is low, it can be regarded as the first service cluster is in an idle state, which may be due to over-configuration of machine resources and not being reasonably utilized.

[0051] Step S206: When the machine resource utilization is greater than the set first utilization threshold, determine at least one second target message topic and a second service cluster corresponding to each second target message topic according to the first number of first preset message topics and the total number of messages.

[0052] Among them, if the machine resource utilization is greater than the set first utilization threshold, for example, the processor utilization exceeds 80%, it can be regarded as the first service cluster is in an overloaded state, and it can be considered to join other service clusters to participate in the message processing of the first preset message topic to balance the load of the first service cluster. In one embodiment, if the first number of the first preset message topics is two, then one of the two first preset message topics can be determined as the second target message topic, and the second service cluster is selected to process the message of the second target message topic, while the first service cluster continues to process another first preset message topic. In one embodiment, if the first number of the first preset message topics is three, and the total amount of messages between different first preset message topics is similar, then it can be considered to randomly select a second target message topic from the three first preset message topics, and select the second service cluster to process the second target message topic, while the first service cluster continues to process the remaining two first preset message topics. In addition, in this embodiment, if there is a first preset message topic in the first preset message topic whose total message volume is close to the sum of the total message volume of the other two first preset message topics, then the two first preset message topics with smaller total message volume among the three first preset message topics can be considered as the second target message topics, and two second service clusters are selected to process the corresponding second target message topics respectively, while the first service cluster continues to process the remaining first preset message topic. Of course, the developer can also specifically set other solutions for diverting to the second service cluster for processing based on the first quantity and the total message volume, which is not limited in this application.

[0053] Step S207: Change the processing execution object in the second cluster configuration information corresponding to each second target message topic to the corresponding second service cluster, so that after the first service cluster and the second service cluster monitor the information change of the second cluster configuration information, the first service cluster cancels the second consumer instance pointing to the second target message topic, and the second service cluster creates a third consumer instance pointing to the corresponding second target message topic, so that the second service cluster can process the message in the second target message topic based on the third consumer instance.

[0054] Among them, since the second target message topic is originally processed by the first service cluster, if it needs to be changed to the second service cluster for processing, the processing execution object in the second cluster configuration information corresponding to each second target message topic can be changed to the corresponding second service cluster, and the second cluster configuration information will be synchronized to the first service cluster and the second service cluster. During the operation process, the first service cluster can monitor the second cluster configuration information corresponding to the second target message topic to determine whether the content of the second cluster configuration information has changed. Among them, the first service cluster can cancel the second consumer instance pointing to the second target message topic by determining that the cluster name in the second cluster configuration information corresponding to the second target message topic does not match its own cluster name, and no longer process the message of the second target message topic. During the operation process, the second service cluster can also monitor the second cluster configuration information corresponding to the second target message topic to determine whether the content of the second cluster configuration information has changed. Among them, the second service cluster can create a third consumer instance pointing to the second target message topic by determining that the cluster name in the second cluster configuration information corresponding to a second target message topic is changed to its own cluster name, and process the message in the second target message topic based on the third consumer instance.

[0055] In one embodiment, Figure 4 A schematic diagram of a process for splitting multiple first preset message topics into different service clusters for message processing is provided in an embodiment of the present application, such as Figure 4 As shown, the multiple first preset message topics 202 originally configured in the first service cluster 201 are respectively split into the first service cluster 201 and the second service cluster 203 for message processing.

[0056] As described above, by determining the second target message topic corresponding to each second service cluster from the first preset message topic, and changing the second cluster configuration information corresponding to the second target message topic, the service cluster of the second target message topic can be adjusted from the first service cluster to the corresponding second service cluster, so that different message topics can be split into different service clusters for processing, and the consumer instances of the first service cluster can be dynamically adjusted to achieve more refined resource management and load balancing. In addition, configuration-controlled consumer creation and execution are introduced, so that the service cluster can create and start consumer instances on demand without hard coding or predefinition, thereby improving the flexibility and scalability of the system, enabling it to quickly respond to changes in business needs and effectively cope with message processing scenarios of different scales and complexities.

[0057] Figure 5 A flowchart of a topic message processing method including a process of configuring a first service cluster to process a second preset message topic is provided in an embodiment of the present application. Figure 5As shown, the topic message processing method includes the following steps:

[0058] Step S301: Obtain the total number of messages of each first preset message topic currently processed by the first service cluster.

[0059] Step S302: When the total amount of first messages of the first target message topic in the first preset message topic is greater than a preset quantity threshold, at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic are determined according to the total amount of first messages.

[0060] Step S303: Change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage.

[0061] Step S304: Change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0062] Step S305: When the total number of messages in each first preset message topic is less than or equal to a preset number threshold, obtain the machine resource utilization of the first service cluster.

[0063] Step S306: When the machine resource utilization is greater than the set first utilization threshold, determine at least one second target message topic and a second service cluster corresponding to each second target message topic according to the first number of first preset message topics and the total number of messages.

[0064] Step S307: Change the processing execution object in the second cluster configuration information corresponding to each second target message topic to the corresponding second service cluster, so that after the first service cluster and the second service cluster monitor the change of the second cluster configuration information, the first service cluster cancels the second consumer instance pointing to the second target message topic, and the second service cluster creates a third consumer instance pointing to the corresponding second target message topic, so that the second service cluster can process the message in the second target message topic based on the third consumer instance.

[0065] Step S308: When the machine resource utilization is less than the set second utilization threshold, the consumer state in the instance configuration information corresponding to the second preset message topic in the unconfigured state is changed to an enabled consumer, and the processing execution object in the third cluster configuration information corresponding to the second preset message topic is changed to the first service cluster, so that after the first service cluster monitors the information changes in the instance configuration information and the third cluster configuration information, a fourth consumer instance pointing to the second preset message topic is created, so that the first service cluster can process the message in the second preset message topic based on the fourth consumer instance.

[0066] Among them, if the machine resource utilization is less than the set second utilization threshold, for example, the processor utilization is less than 30%, it can be regarded as the first service cluster is in an idle state, and it can be considered to increase the message topic that the first service cluster needs to process to avoid the waste of resources of the first service cluster. The second preset message topic in the unconfigured state can be a message topic to be assigned to the service cluster for message processing. By changing the consumer state in the instance configuration information corresponding to the second preset message topic to enable the consumer, the second preset message topic can be configured as a service cluster that can create a consumer instance to process its message, and the processing execution object in the third cluster configuration information corresponding to the second preset message topic is changed to the first service cluster, which is equivalent to configuring the first service cluster to process the message on the second preset message topic. After completing the adjustment of the instance configuration information and the third cluster configuration information, the instance configuration information and the third cluster configuration information can be synchronized to the first service cluster. Then, during the operation, the first service cluster can monitor the instance configuration information and the third cluster configuration information to determine whether the content of the third cluster configuration information has changed. Among them, the first service cluster can create a fourth consumer instance pointing to the second preset message topic and process the message in the second preset message topic based on the fourth consumer instance by determining that the second preset message topic is in an enabled consumer state and the cluster name in the third cluster configuration information corresponding to the second preset message topic is its own cluster name.

[0067] As described above, by configuring the second preset message topic to enable the consumer and the corresponding processing execution object to be the first service cluster when the machine resource utilization is less than the set second utilization threshold, the consumer instance of the first service cluster can be dynamically adjusted to adapt to the change of the message topic, thereby improving the resource utilization of the first service cluster and making better use of the idle machine resources of the first service cluster.

[0068] Figure 6 A flowchart of a topic message processing method including a process of determining a sub-message topic and a corresponding sub-service cluster is provided in an embodiment of the present application. Figure 6 As shown, the topic message processing method includes the following steps:

[0069] Step S401, obtain the total number of messages of each first preset message topic currently processed by the first service cluster. When the total number of first messages of the first target message topic in the first preset message topic is greater than the preset number threshold, compare the total number of first messages with the set multiple preset number intervals, determine the target number interval in which the total number of first messages is located from the multiple preset number intervals, query the number of sub-message topics corresponding to the target number interval, determine at least one sub-message topic that matches the number of sub-message topics from the set candidate sub-message topics, and assign a corresponding sub-service cluster to each sub-message topic based on the set candidate sub-service clusters.

[0070] Among them, the preset number interval can be a different numerical range corresponding to the total amount of messages divided in advance. The larger the numerical range corresponding to the preset number interval where the total amount of the first message is located, the larger the total amount of messages is, and more sub-message topics can be configured accordingly to achieve a better fragmentation processing effect. For example, the preset number interval includes a first number interval, a second number interval, and a third number interval, and the first number interval is smaller than the second number interval, and the second number interval is smaller than the third number interval. The number of sub-message topics corresponding to the first number interval can be set to 2, the number of sub-message topics corresponding to the second number interval can be set to 3, and the number of sub-message topics corresponding to the third number interval can be set to 4. Of course, the setting of the number of sub-message topics is only an exemplary description. Developers can also make adaptive adjustments according to the load balancing strategy of the actual application scenario and the number of allocable sub-message topics, which is not limited in this application. To determine the number of sub-message topics, at least one sub-message topic that matches the number of sub-message topics can be determined from the candidate sub-message topics, and the candidate sub-message topic is a plurality of pre-set idle message topics. Finally, each sub-message topic can be assigned a corresponding sub-service cluster to achieve the purpose of splitting the message into different service clusters for processing.

[0071] Step S402: Change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage.

[0072] Step S403: Change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0073] As described above, by determining at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic, the messages of the first target message topic can be effectively split into different sub-consumption topics without being restricted by the number of partitions of the first target message topic, thereby achieving the purpose of horizontal capacity expansion, adding a sub-service cluster to process messages of the first target message topic, improving business message processing capabilities, and parallel processing improves the overall throughput of the system.

[0074] Figure 7 A flowchart of a topic message processing method including a process of determining a second target message topic and a corresponding second service cluster is provided in an embodiment of the present application. Figure 7 As shown, the topic message processing method includes the following steps:

[0075] Step S501: Obtain the total number of messages of each first preset message topic currently processed by the first service cluster.

[0076] Step S502: When the total amount of first messages of the first target message topic in the first preset message topic is greater than a preset quantity threshold, at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic are determined according to the total amount of first messages.

[0077] Step S503: Change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage.

[0078] Step S504: Change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0079] Step S505: When the total amount of messages in each first preset message topic is less than or equal to a preset quantity threshold, obtain the machine resource utilization of the first service cluster.

[0080] Step S506: When the machine resource utilization is greater than the set first utilization threshold, determine the number of target message topics based on the first number of first preset message topics and the total number of messages, determine at least one second target message topic that matches the number of target message topics from the first preset message topics, and assign a corresponding second service cluster to each second target message topic based on the set candidate service cluster.

[0081] Among them, the target number of message topics that need to be split out and processed by other service clusters can be determined based on the first number of the first preset message topics and the total number of messages. For example, if the first number is two, then the number of target message topics can be determined to be one. For another example, if the first number is three and the total number of messages of different first preset message topics is similar, the number of target message topics can be selected to be one or two, which can alleviate the load condition of the first service cluster to varying degrees. The number of target message topics can be determined as the number of second target message topics, and at least one second target message topic that matches the number of target message topics can be further determined from the first preset message topic. Finally, a corresponding second service cluster is assigned to each second target message topic based on the set candidate service cluster. The candidate service cluster can be a service cluster in an idle state, so as to achieve the purpose of splitting multiple message topics to different service clusters for message processing.

[0082] Step S507: Change the processing execution object in the second cluster configuration information corresponding to each second target message topic to the corresponding second service cluster, so that after the first service cluster and the second service cluster monitor the change of the second cluster configuration information, the first service cluster cancels the second consumer instance pointing to the second target message topic, and the second service cluster creates a third consumer instance pointing to the corresponding second target message topic, so that the second service cluster can process the message in the second target message topic based on the third consumer instance.

[0083] As described above, by determining at least one second target message topic and a second service cluster corresponding to each second target message topic, different message topics can be effectively split into different clusters for message processing, service clusters can be reasonably divided, resources can be used more efficiently, and message processing can be parallelized and load balancing can be achieved, greatly improving the overall consumption performance of the system.

[0084] Figure 8 This is a structural block diagram of a topic message processing device provided in an embodiment of the present application. The device is configured to execute the topic message processing method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. Figure 8 As shown, the device comprises:

[0085] The topic cluster determination module 301 is configured to obtain the total amount of messages of each first preset message topic currently processed by the first service cluster, and when the total amount of first messages of the first target message topic in the first preset message topic is greater than a preset quantity threshold, determine at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic according to the total amount of the first messages;

[0086] The first configuration information adjustment module 302 is configured to change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, the message of the first target message topic is forwarded to the corresponding sub-message topic for storage;

[0087] The second configuration information adjustment module 303 is configured to change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

[0088] In the above, by obtaining the total number of messages of each first preset message topic currently processed by the first service cluster, when the total number of first messages of the first target message topic in the first preset message topic is greater than the preset number threshold, at least one sub-message topic and a sub-service cluster corresponding to each sub-message topic are determined according to the total number of first messages; the message processing mode in the consumption configuration information corresponding to the first target message topic is changed to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the messages of the first target message topic to the corresponding sub-message topic for storage; the processing execution object in the first cluster configuration information corresponding to each sub-message topic is changed to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change of the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the messages in the sub-message topic based on the first consumer instance. In the above scheme, by comparing the total amount of messages of each first preset message topic with the preset quantity threshold, the message load of each preset message topic can be effectively judged, and by determining at least one sub-message topic and the sub-service cluster corresponding to each sub-message topic according to the total amount of the first messages, and adjusting the consumption configuration information corresponding to the first target message topic and the current cluster name in the first cluster configuration information corresponding to each sub-message topic in turn, the pending messages of the high-load message topic can be effectively configured to be forwarded to multiple sub-message topics, and the corresponding multiple sub-service clusters can be configured for parallel processing, so as to achieve a more balanced load distribution, avoid the performance bottleneck caused by overload of a single service cluster, and improve the message processing speed in real-time large-scale message processing scenarios, and also can flexibly create consumer instances to quickly adapt to changing business needs, effectively improving the efficiency and dynamic response capabilities of message processing.

[0089] In a possible embodiment, a third configuration information adjustment module is further included, configured to:

[0090] When the total amount of messages of each first preset message topic is less than or equal to a preset quantity threshold, obtaining the machine resource utilization rate of the first service cluster;

[0091] When the machine resource utilization is greater than a set first utilization threshold, determining at least one second target message topic and a second service cluster corresponding to each second target message topic according to the first number of the first preset message topics and the total number of messages;

[0092] Change the processing execution object in the second cluster configuration information corresponding to each second target message topic to the corresponding second service cluster, so that after the first service cluster and the second service cluster monitor the information change in the second cluster configuration information, the first service cluster cancels the second consumer instance pointing to the second target message topic, and the second service cluster creates a third consumer instance pointing to the corresponding second target message topic, so that the second service cluster can process the message in the second target message topic based on the third consumer instance.

[0093] In a possible embodiment, a fourth configuration information adjustment module is further included, configured to:

[0094] When the machine resource utilization is less than the set second utilization threshold, the consumer state in the instance configuration information corresponding to the second preset message topic in the unconfigured state is changed to an enabled consumer, and the processing execution object in the third cluster configuration information corresponding to the second preset message topic is changed to the first service cluster, so that after the first service cluster monitors the information changes in the instance configuration information and the third cluster configuration information, a fourth consumer instance pointing to the second preset message topic is created, so that the first service cluster can process the message in the second preset message topic based on the fourth consumer instance.

[0095] In a possible embodiment, the first configuration information adjustment module 302 is further configured to:

[0096] An enable forwarding flag is set in the consumption configuration information corresponding to the first target message topic, and a preset forwarding rule and a topic name of each sub-message topic are added to the consumption configuration information.

[0097] In a possible embodiment, the second configuration information adjustment module 303 is further configured to:

[0098] The current cluster name set in the first cluster configuration information corresponding to each sub-message topic is modified to the cluster name of the corresponding sub-service cluster.

[0099] In a possible embodiment, the topic cluster determination module 301 is further configured to:

[0100] Compare the total amount of the first message with a plurality of preset quantity intervals, determine a target quantity interval in which the total amount of the first message is located from the plurality of preset quantity intervals, and query the number of sub-message topics corresponding to the target quantity interval;

[0101] Determine at least one sub-message topic that matches the number of sub-message topics from the set candidate sub-message topics;

[0102] A corresponding sub-service cluster is allocated to each sub-message topic based on the set candidate sub-service clusters.

[0103] In a possible embodiment, the third configuration information adjustment module is further configured to:

[0104] Determine the number of target message topics according to the first number of first preset message topics and the total number of messages;

[0105] Determine at least one second target message topic that matches the number of target message topics from the first preset message topics;

[0106] A corresponding second service cluster is allocated to each second target message topic based on the set candidate service cluster.

[0107] Fig. 9 A structural diagram of a topic message processing device provided in an embodiment of the present application, such as Fig. 9 As shown, the device includes a processor 401, a memory 402, an input device 403 and an output device 404; the number of processors 401 in the device can be one or more. Fig. 9 A processor 401 is taken as an example; the processor 401, memory 402, input device 403 and output device 404 in the device can be connected by a bus or other means. Fig. 9 The example of connecting through a bus is taken. The memory 402, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the subject message processing method in the embodiment of the present application. The processor 401 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 402, that is, implements the above-mentioned subject message processing method. The input device 403 can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 404 may include a display device such as a display screen.

[0108] An embodiment of the present application also provides a non-volatile storage medium containing computer executable instructions, which, when executed by a computer processor, is configured to execute a topic message processing method described in the above embodiment, which includes: obtaining the total amount of messages of each first preset message topic currently processed by the first service cluster, and when the total amount of first messages of the first target message topic in the first preset message topic is greater than the preset quantity threshold, determining at least one sub-message topic and the sub-service cluster corresponding to each sub-message topic according to the total amount of the first messages; changing the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage; changing the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change of the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster processes the message in the sub-message topic based on the first consumer instance.

[0109] It is worth noting that in the embodiment of the above-mentioned topic message processing device, the various units and modules included are divided only according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the embodiments of the present application.

[0110] In some possible implementations, various aspects of the method provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is configured to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the topic message processing method recorded in the embodiment of the present application. The program product may be implemented in any combination of one or more readable media.

Claims

1. A topic message processing method, characterized in that: include: Obtain the total number of messages of each first preset message topic currently processed by the first service cluster, and when the total number of first messages of the first target message topic in the first preset message topic is greater than a preset number threshold, determine at least one sub-message topic and a sub-service cluster corresponding to each of the sub-message topics according to the first total number of messages; Change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, it forwards the message of the first target message topic to the corresponding sub-message topic for storage; The processing execution object in the first cluster configuration information corresponding to each of the sub-message topics is changed to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change in the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

2. The topic message processing method according to claim 1, characterized in that: Also includes: When the total amount of messages of each first preset message topic is less than or equal to the preset quantity threshold, obtaining the machine resource utilization rate of the first service cluster; When the machine resource utilization is greater than a set first utilization threshold, determining at least one second target message topic and a second service cluster corresponding to each second target message topic according to the first number of the first preset message topics and the total number of messages; Change the processing execution object in the second cluster configuration information corresponding to each second target message topic to the corresponding second service cluster, so that after the first service cluster and the second service cluster monitor the information change of the second cluster configuration information, the first service cluster cancels the second consumer instance pointing to the second target message topic, and the second service cluster creates a third consumer instance pointing to the corresponding second target message topic, so that the second service cluster can process the message in the second target message topic based on the third consumer instance.

3. The topic message processing method according to claim 2, characterized in that: Also includes: When the machine resource utilization is less than the set second utilization threshold, the consumer state in the instance configuration information corresponding to the second preset message topic in the unconfigured state is changed to an enabled consumer, and the processing execution object in the third cluster configuration information corresponding to the second preset message topic is changed to the first service cluster, so that after the first service cluster monitors the information changes between the instance configuration information and the third cluster configuration information, a fourth consumer instance pointing to the second preset message topic is created, so that the first service cluster can process the message in the second preset message topic based on the fourth consumer instance.

4. The method for processing topic messages according to any one of claims 1 to 3, characterized in that: The changing the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing includes: An enable forwarding flag is set in the consumption configuration information corresponding to the first target message topic, and a preset forwarding rule and a topic name of each sub-message topic are added to the consumption configuration information.

5. The method for processing topic messages according to any one of claims 1 to 3, characterized in that: The step of changing the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster includes: The current cluster name set in the first cluster configuration information corresponding to each of the sub-message topics is modified to the cluster name of the corresponding sub-service cluster.

6. The topic message processing method according to claim 1, characterized in that: The determining, according to the first message total amount, at least one sub-message topic and a sub-service cluster corresponding to each of the sub-message topics includes: Compare the total amount of the first message with a plurality of preset quantity intervals, determine a target quantity interval in which the total amount of the first message is located from the plurality of preset quantity intervals, and query the number of sub-message topics corresponding to the target quantity interval; Determine at least one sub-message topic that matches the number of sub-message topics from the set candidate sub-message topics; A corresponding sub-service cluster is allocated to each of the sub-message topics based on the set candidate sub-service cluster.

7. The topic message processing method according to claim 2, characterized in that: The determining, according to the first quantity of the first preset message topics and the total amount of messages, at least one second target message topic and a second service cluster corresponding to each second target message topic includes: Determine the number of target message topics according to the first number of the first preset message topics and the total number of messages; Determine at least one second target message topic that matches the number of the target message topics from the first preset message topics; A corresponding second service cluster is allocated to each of the second target message topics based on the set candidate service cluster.

8. A topic message processing device, characterized in that: include: a topic cluster determination module, configured to obtain the total amount of messages of each first preset message topic currently processed by the first service cluster, and determine at least one sub-message topic and a sub-service cluster corresponding to each of the sub-message topics according to the first total amount of messages when the total amount of first messages of the first target message topic in the first preset message topic is greater than a preset quantity threshold; A first configuration information adjustment module is configured to change the message processing mode in the consumption configuration information corresponding to the first target message topic to forwarding processing, so that after the first service cluster monitors the information change of the consumption configuration information, the message of the first target message topic is forwarded to the corresponding sub-message topic for storage; The second configuration information adjustment module is configured to change the processing execution object in the first cluster configuration information corresponding to each sub-message topic to the corresponding sub-service cluster, so that after the sub-service cluster monitors the information change of the first cluster configuration information, it creates a first consumer instance pointing to the corresponding sub-message topic, so that the sub-service cluster can process the message in the sub-message topic based on the first consumer instance.

9. A topic message processing device, the device comprising: one or more processors; A storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the topic message processing method described in any one of claims 1-7.

10. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the topic message processing method according to any one of claims 1 to 7 when executed by a computer processor.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the topic message processing method described in any one of claims 1 to 7 is implemented.

Images