Message processing method, apparatus, device, storage medium, and program product

Abstract

This application provides a message processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, comprising: obtaining topic information corresponding to a consumer group, wherein the consumer group includes at least one consumer instance of messages; when the topic information includes a second topic with a higher priority than a first topic, determining a first consumer instance in the consumer group that needs to be allocated messages, wherein the first topic is a topic subscribed to by the consumer instance in the consumer group; obtaining consumption information of the first consumer instance, the consumption information indicating the topic of the messages that the first consumer instance needs to process; and allocating messages of the second topic to the first consumer instance based on the consumption information of the first consumer instance. This improves the convenience and efficiency of managing consumers.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a message processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In the message processing of related technologies, when managing multiple consumers, whether server-side partitioning or client-side partitioning is used, it is a decentralized management model. That is, the management process for multiple consumers in a consumer group is implemented by each consumer itself. At the same time, the management process for a single consumer must be stopped before the management process for that consumer can be executed. This makes the consumer management process more complex and less efficient. Summary of the Invention

[0003] This application provides a message processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the convenience and efficiency of managing consumers.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a message processing method, including:

[0006] Obtain the topic information corresponding to the consumer group, wherein the consumer group includes at least one consumer instance of the message;

[0007] When the topic information includes a second topic with a higher priority than the first topic, the first consumer instance in the consumer group that needs to be allocated messages is determined, wherein the first topic is the topic subscribed to by the consumer instance in the consumer group;

[0008] Obtain the consumption information of the first consumer instance, wherein the consumption information is used to indicate the subject of the message that the first consumer instance needs to process;

[0009] Based on the consumption information of the first consumer instance, the message of the second topic is assigned to the first consumer instance.

[0010] This application provides a message processing apparatus, including:

[0011] The first acquisition module is used to acquire topic information corresponding to a consumer group, wherein the consumer group includes at least one consumer instance of the message;

[0012] The determination module is used to determine the first consumer instance in the consumer group that needs to be allocated messages when the topic information includes a second topic with a higher priority than the first topic, wherein the first topic is the topic subscribed to by the consumer instance in the consumer group;

[0013] The second acquisition module is used to acquire the consumption information of the first consumer instance, wherein the consumption information is used to indicate the topic of the message that the first consumer instance needs to process;

[0014] The allocation module is used to allocate messages of the second topic to the first consumer instance based on the consumption information of the first consumer instance.

[0015] In the above scheme, the determining module is further configured to obtain a first consumption identifier for each consumer instance in the consumer group, and obtain a second consumption identifier for the consumer group; the first consumption identifier is used to indicate the topic of the message that the consumer instance can consume, and the second consumption identifier is used to indicate the first consumption identifier of a second consumer instance, wherein the second consumer instance is a consumer instance in the consumer group that can consume messages on the second topic; based on the first consumption identifier and the second consumption identifier, the first consumer instance is selected from the consumer group.

[0016] In the above scheme, the determining module is further configured to obtain the mapping relationship between the first consumption identifier and the topic set; based on the mapping relationship, determine the topic set corresponding to different first consumption identifiers, wherein the topic set includes at least one topic; determine the target topic set including the second topic from the topic sets corresponding to each first consumption identifier, and determine the first consumption identifier corresponding to the target topic set as the second consumption identifier.

[0017] In the above scheme, the determining module is further configured to analyze each consumer instance to obtain the topics of messages that the consumer instance can consume; cluster the topics of messages that each consumer instance can consume to obtain at least one cluster, and the topics in each cluster constitute a topic set; generate a cluster identifier that corresponds one-to-one with the cluster, use the cluster identifier as the first consumer identifier, and establish a mapping relationship between the first consumer identifier and the topic set.

[0018] In the above scheme, the number of second consumer identifiers is at least one. The determining module is further configured to match the corresponding first consumer identifier with each second consumer identifier for each consumer instance in the consumer group to obtain a matching result. When the matching result indicates that there is a second consumer identifier that matches the first consumer identifier, the corresponding consumer instance is determined as the first consumer instance.

[0019] In the above scheme, the topic information is included in the consumption management information. The determining module is further configured to, when the consumption management information includes a disable identifier, filter out a second consumer instance that does not need to be allocated a message from the consumer group based on the disable identifier; wherein, the disable identifier is the instance identifier of the consumer instance that does not need to be allocated a message; and determine the first consumer instance that needs to be allocated a message based on the consumer instances in the consumer group other than the second consumer instance.

[0020] In the above scheme, the determining module is further configured to obtain the instance identifier of each consumer instance in the consumer group; match the disabled identifier with the instance identifier of each consumer instance to obtain a matching result; when the matching result indicates that the instance identifier matches the disabled identifier, the consumer instance corresponding to the instance identifier is taken as the second consumer instance; and unmount the partition historically allocated to the second consumer instance.

[0021] In the above scheme, the topic information is included in the consumption management information, and the allocation module is further configured to use the partition information to allocate the message of the second topic to the first consumer instance when the consumption management information includes partition information; wherein, the partition information is used to indicate the partition allocated to each of the first consumer instances.

[0022] In the above scheme, the device further includes a third acquisition module, which is used to acquire instance information of each consumer instance in the consumer group; wherein the instance information includes at least one of the following: the instance identifier of the consumer instance, the group identifier of the consumer group, the topic subscribed by the consumer instance, status information indicating whether the consumer instance is consuming normally, and partition information; and generates a consumption report based on the instance information of each consumer; wherein the consumption report is used to view the consumption status of each consumer instance.

[0023] In the above scheme, the first acquisition module is further configured to, in response to receiving consumption management information, parse the consumption management information to obtain topic information corresponding to the consumer group, wherein the consumption management information is configured by the manager of the consumer group; or, obtain the pre-set topic information from the storage structure.

[0024] In the above scheme, the consumption management information also includes a disable identifier and partition information. The disable identifier is the instance identifier of a consumer instance that does not need to be allocated messages, and the partition information is used to indicate the partition allocated to each consumer instance. The determining module is further configured to, based on the disable identifier, filter out a third consumer instance that cannot process messages from the consumer group; designate the consumer instances other than the third consumer instance among the at least one consumer instance as fourth consumer instances; obtain a first consumption identifier corresponding to each fourth consumer instance, and obtain a second consumption identifier of the consumer group; filter out the first consumer instance from the fourth consumer instances based on the first consumption identifier and the second consumption identifier; the allocation module 4554 is further configured to, using the partition information, allocate messages of the second topic to the first consumer instance.

[0025] In the above scheme, the device further includes a fourth acquisition module, which is used to acquire the load information of each consumer instance in the consumer group when the topic information only includes the first topic; based on the load information, select a fifth consumer instance from the consumer group whose load is less than or equal to the load threshold; and based on the consumption information of the fifth consumer instance, allocate the message of the first topic to the fifth consumer instance.

[0026] In the above scheme, there are multiple consumer groups, and the device further includes a sorting module, which is used to sort the multiple consumer groups to obtain a consumer group sequence; the first acquisition module is also used to determine the topic information corresponding to the consumer group when the processing time point of the consumer group arrives based on the consumer group sequence.

[0027] This application provides an electronic device, including:

[0028] Memory is used to store executable instructions or computer programs.

[0029] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the message processing method provided in the embodiments of this application.

[0030] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the message processing method provided in this application.

[0031] This application provides a computer program product including computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the message processing method provided in this application.

[0032] The embodiments of this application have the following beneficial effects:

[0033] When the acquired topic information indicates that a second topic with higher priority than the first topic exists within the consumer group, the first consumer instance in the consumer group that needs to be allocated a message partition is determined. Then, the consumption information of the first consumer instance is obtained, and based on this information, messages from the second topic are allocated to the first consumer instance. Thus, after obtaining the topic information for managing the consumer group, a centralized management model is adopted, managing consumers within the group based on topic information. Compared to the decentralized management model in related technologies, this simplifies the management process and improves the convenience and efficiency of managing consumers. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the architecture of the message processing system 100 provided in an embodiment of this application;

[0035] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0036] Figure 3 This is a flowchart illustrating the message processing method provided in an embodiment of this application;

[0037] Figure 4 This is a schematic diagram illustrating relevant information about consumer instances provided in the embodiments of this application;

[0038] Figure 5 This is a schematic diagram of the process for determining the first consumer instance provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of the mapping relationship provided in the embodiments of this application;

[0040] Figure 7 This is a schematic diagram of the second consumer identifier provided in an embodiment of this application;

[0041] Figure 8 This is a schematic diagram illustrating the process of screening the first consumer instance provided in an embodiment of this application;

[0042] Figure 9 This is a schematic diagram illustrating the process of screening the first consumer instance provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of a consumption report provided in an embodiment of this application;

[0044] Figure 11 This is a schematic diagram of consumer-related information in the related technologies provided in the embodiments of this application;

[0045] Figure 12 This is a schematic diagram of consumer-related information in the related technologies provided in the embodiments of this application;

[0046] Figure 13 This is a schematic diagram illustrating the process of managing consumer groups provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0051] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0052] 1) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0053] 2) Client, also known as user terminal, refers to the program that provides local services to users in contrast to the server. Except for some applications that can only run locally, it is generally installed on the terminal and needs to work in conjunction with the server. That is, there needs to be a corresponding server and service program on the network to provide the corresponding services. Thus, a specific communication connection needs to be established between the client and the server to ensure the normal operation of the application, such as virtual scene clients (such as game clients) and video clients.

[0054] 3) Artificial Intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0055] 4) A topic is a set of messages that is used to receive messages sent by producers and for consumers to subscribe to so that consumers can consume messages in the subscribed topic.

[0056] 5) Producer: A component in the message queue system, responsible for creating and sending messages. It is usually connected to the business system and is used to encapsulate business data into message format and send it to the message queue system.

[0057] 6) A consumer, or consumer instance, is a component in a message queue system that receives messages from the message queue system and consumes or processes them.

[0058] 7) Message Queue (MQ): An asynchronous cross-process communication mechanism used for upstream and downstream message passing.

[0059] 8) Broker, also known as a proxy node, is a component in a message queue system responsible for storing and retrieving messages.

[0060] 9) Master, also known as the master node, is the cluster management component in the message queue system. It is used to synchronize metadata with producers and consumers and coordinate the Broker to complete the message processing process based on the message queue.

[0061] 10) Coordinator, also known as the coordinating node, is a service component in a message queue system. It is responsible for coordinating the interaction between multiple components or consumers in a distributed system. In other words, it is used to maintain the metadata of the consumer group, including information about consumer instances and the partition allocation of consumers.

[0062] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the message processing system 100 provided in this application embodiment. The terminal (terminal 400 is shown as an example) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using wireless or wired links.

[0063] Terminal 400 is used to receive topic information corresponding to the consumer group set by the consumer group manager, and send the topic information corresponding to the consumer group to the server.

[0064] Server 200 is used to: obtain topic information corresponding to a consumer group, wherein the consumer group includes at least one consumer instance of a message; when the topic information includes a second topic with a higher priority than the first topic, determine the first consumer instance in the consumer group that needs to be allocated a message, wherein the first topic is the topic subscribed to by the consumer instance in the consumer group; obtain the consumption information of the first consumer instance, the consumption information being used to indicate the topic of the message that the first consumer instance needs to process; and allocate the message of the second topic to the first consumer instance based on the consumption information of the first consumer instance.

[0065] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, set-top box, smart voice interaction device, smart home appliance, virtual reality device, vehicle terminal, aircraft, portable music player, personal digital assistant, dedicated messaging device, portable gaming device, smart speaker, and smartwatch, but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0066] The electronic device implementing the message processing method provided in the embodiments of this application will now be described. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be a server or a terminal. The electronic device is used as an example. Figure 1 Taking the server shown as an example, Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0067] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0068] User interface 430 includes one or more output devices 431 that enable the display of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0069] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0070] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0071] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0072] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0073] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0074] Presentation module 453 is configured to enable the display of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0075] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0076] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A message processing device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a first acquisition module 4551, a determination module 4552, a second acquisition module 4553, and an allocation module 4554. These modules are logically linked and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0077] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the message processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the message processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0078] In some embodiments, the terminal or server can implement the message processing method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a local client, which is a program that needs to be installed in the operating system to run, such as an instant messaging APP or a web browser APP; it can also be a mini-program, i.e., a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of client, module, or plugin.

[0079] Based on the above description of the message processing system and electronic device provided in the embodiments of this application, the message processing method provided in the embodiments of this application is described below. In actual implementation, the message processing method provided in the embodiments of this application can be implemented by a terminal or a server alone, or by a terminal and a server working together, so that... Figure 1 The following description uses the example of server 200 executing the message processing method provided in this embodiment of the application independently. See also... Figure 3 , Figure 3 This is a flowchart illustrating the message processing method provided in the embodiments of this application. Next, it will be discussed in conjunction with... Figure 3 The steps shown are explained.

[0080] Step 101: The server obtains the topic information corresponding to the consumer group, wherein the consumer group includes at least one consumer instance of the message.

[0081] In practice, the topic information corresponding to a consumer group can be cached in the storage structure or configured in real time through APIs, tools, configuration files, etc. Specifically, the process of obtaining the topic information corresponding to a consumer group can be as follows: in response to receiving consumer management information, the consumer management information is parsed to obtain the topic information corresponding to the consumer group. The consumer management information is configured by the administrator of the consumer group and is used to manage the consumer group; or, the pre-set topic information can be obtained from the storage structure.

[0082] It should be noted that the topic information cached in the storage structure can be set when the consumer instance is registered. Specifically, the registration request for the consumer instance is received, the registration request is parsed to obtain the instance identifier of the consumer instance, the group identifier of the consumer group to which it belongs, the subscribed topic, and the corresponding consumer identifier; then, based on the subscribed topic, topic information is generated and stored in the storage structure.

[0083] Here, the parsed instance identifier indicates the consumer instance, the group identifier indicates the consumer group to which the corresponding consumer instance belongs (which can be the group name), the subscribed topic indicates the message the corresponding consumer instance will consume, and the consumption identifier indicates the topic of the message that the corresponding consumer instance can consume; for example, see Figure 4 , Figure 4 This is a schematic diagram illustrating relevant information about consumer instances provided in the embodiments of this application, based on... Figure 4 The first column, which is also the column containing the consumer instance, is the instance identifier of the consumer instance. The second column is the consumer group name, which is also the group identifier of the consumer group to which the consumer instance belongs. The third column is the topic subscribed to by the consumer instance. The fourth column is the consumption identifier of the consumer instance. If the consumption identifiers are the same, it indicates that the corresponding consumer instances have the same consumption capacity, that is, they can consume messages with the same topics.

[0084] It should be noted that the consumption management information for a consumer group can be obtained during the consumption process of at least one consumer instance. For example, after a consumer instance registers, messages in the topics subscribed to by the consumer instance are assigned to the corresponding consumer instance so that the consumer instance can consume the messages. During the consumption process of the consumer instance, the consumer group's consumption management information is received.

[0085] It should be noted that a consumer instance, or consumer, is used to consume messages on a corresponding topic. Multiple consumer instances form a consumer group to jointly consume one or more topics. Consumer instances in each consumer group share messages. Each message is sent to only one consumer instance in the consumer group, thus ensuring that a message is not processed multiple times by multiple consumer instances within the same consumer group.

[0086] Step 102: When the topic information includes a second topic with a higher priority than the first topic, determine the first consumer instance in the consumer group that needs to be allocated messages, where the first topic is the topic subscribed to by the consumer instance in the consumer group.

[0087] It should be noted that there can be one or more first topics, and one or more second topics. The first topic is the topic subscribed to by the consumer instances in the consumer group, while the second topic is the topic subscribed to by a specified consumer group. Specifying the topic subscribed to by a specified consumer group means that the consumer instances in the consumer group must consume the corresponding topic. At the same time, the priority here refers to the consumption priority, that is, the execution priority. It means that when only the first topic exists, topic subscription and consumption are performed according to the first topic; when the second topic exists, the consumer group subscribes to the topic set based on the second topic and performs the corresponding consumption processing.

[0088] The process involves several steps. After obtaining the topic information, it is parsed. If the parsing result indicates that the topic information includes a second topic with a higher priority than the first topic, the first consumer instance in the consumer group that needs to be allocated messages is determined. If the parsing result indicates that the topic information only includes the first topic, the fifth consumer instance in the consumer group that needs to be allocated messages is determined. In subsequent processes, based on the consumption information of the fifth consumer instance, messages related to the first topic are allocated to the fifth consumer instance. The following sections will explain the cases where the topic information includes both the first and second topics, and the cases where the topic information only includes the first topic.

[0089] In some embodiments, when the topic information includes a first topic and a second topic (i.e., a second topic with a higher priority than the first topic), the process for determining the first consumer instance in the consumer group that needs to be allocated messages is described in [reference needed]. Figure 5 , Figure 5 This is a flowchart illustrating the process of determining a first consumer instance provided in an embodiment of this application, based on... Figure 5 When the topic information includes a second topic with a higher priority than the first topic, the process of determining the first consumer instance in the consumer group that needs to be allocated messages can be achieved through the following steps.

[0090] Step 1021: Obtain the first consumer identifier for each consumer instance in the consumer group, and obtain the second consumer identifier for the consumer group.

[0091] It should be noted that the first consumption identifier is used to indicate the topic of the message that the consumer instance can consume, and the second consumption identifier is used to indicate the first consumption identifier that the second consumer instance has. The second consumer instance is the consumer instance in the consumer group that can consume messages on the second topic.

[0092] It should be noted that each consumer instance in the consumer group has a corresponding consumption identifier, which indicates the consumption capability of the corresponding consumer instance, that is, the topics that the consumer instance can consume. When it is determined that the topic information includes a second topic, since not every consumer instance in the consumer group may be able to consume or process the second topic, it is necessary to first obtain the consumption identifier of the consumer instance in the consumer group that can consume messages of the second topic, that is, the consumption identifier of the second consumer instance. In the subsequent process, based on the consumption identifier of the second consumer instance and the consumption identifier of each consumer instance, the first consumer instance that needs to be allocated messages, that is, the first consumer instance that can consume messages of the second topic, is selected.

[0093] In practice, the process of obtaining the second consumer identifier of a consumer group can be as follows: obtaining the mapping relationship between the first consumer identifier and the topic set; based on the mapping relationship, determining the topic set corresponding to different first consumer identifiers, wherein the topic set includes at least one topic; determining the target topic set including the second topic from the topic sets corresponding to each first consumer identifier, and determining the consumer identifier corresponding to the target topic set as the second consumer identifier.

[0094] In practice, the mapping relationship between the first consumer identifier and the topic set can be either a pre-built mapping relationship or a real-time built mapping relationship. Specifically, the process of obtaining the mapping relationship between the first consumer identifier and the topic set can be as follows: analyze each consumer instance to obtain the topics of messages that the consumer instance can consume; cluster the topics of messages that each consumer instance can consume to obtain at least one cluster, and the topics in each cluster constitute a topic set; generate a cluster identifier that corresponds one-to-one with the cluster, use the cluster identifier as the first consumer identifier, and establish the mapping relationship between the first consumer identifier and the topic set.

[0095] It should be noted that the topics that a consumer instance can consume can be the same as or different from the topics that the consumer subscribes to. The topics that a consumer instance can consume include the topics that the consumer subscribes to. That is, a consumer instance needs to select some topics to subscribe to from at least one of the topics that it can consume.

[0096] The process of clustering the topics of messages that each consumer instance can consume to obtain at least one cluster is as follows: For example, there are four consumer instances, namely consumer instance A, consumer instance B, consumer instance C and consumer instance D. Consumer instance A can consume messages with topics X, Y and Z. Consumer instance B can process topics X and Z. Consumer instance C can process topics X and Z. Consumer instance D can also process topics X and Y. Then, these four topics are clustered to obtain three clusters, namely cluster X, Y and Z, cluster X and Y, and cluster X and Z. The topics in each cluster constitute a topic set, namely topic set 1 including X, Y and Z, topic set 2 including X and Y, and topic set 3 including X and Z.

[0097] Then, cluster identifiers corresponding one-to-one with the clusters are generated, and the cluster identifiers are used as the first consumption identifiers. A mapping relationship between the first consumption identifiers and the topic set is established. Here, the cluster identifiers and consumption identifiers can be characters or regular expressions, etc. This application does not limit this. Continuing with the above example, cluster identifier V2 can be generated for clusters X, Y, and Z, cluster identifier V0 can be generated for clusters X and Y, and cluster identifier V1 can be generated for clusters X and Z. Thus, the cluster identifiers are used as the first consumption identifiers, and a mapping relationship between the first consumption identifiers and the topic set is established.

[0098] For example, see Figure 6 , Figure 6 This is a schematic diagram of the mapping relationship provided in the embodiments of this application, based on Figure 6 The first column represents the topic sets, namely topic set 1 including X, Y, and Z, topic set 2 including X and Y, and topic set 3 including X and Z; the second column represents the consumption identifiers, namely V1, V2, and V0, where topic set 1 corresponds to V2, topic set 2 corresponds to V0, and topic set 3 corresponds to V1.

[0099] In actual implementation, after obtaining the mapping relationship between the first consumption identifier and the topic set, the topic set corresponding to different first consumption identifiers is determined based on the mapping relationship, and the topic set includes at least one topic; from the topic set corresponding to each first consumption identifier, the target topic set including the second topic is determined, and the first consumption identifier corresponding to the target topic set is determined as the second consumption identifier;

[0100] Following the example above, after obtaining the mapping relationship between the first consumption identifier and the topic set, the topic sets corresponding to different first consumption identifiers are determined based on the mapping relationship. That is, the topic sets corresponding to the first consumption identifier V0 are determined to be X and Y, the topic sets corresponding to the first consumption identifier V1 are X and Z, and the topic sets corresponding to the first consumption identifier V2 are X, Y, and Z. Then, taking the second topic as X and Z as an example, when the second topic is X and Z, the target topic set including the second topic is determined from the topic sets corresponding to each first consumption identifier. That is, the target topic set including the second topic X and Z is determined from the topic sets X and Y, X and Z, and X, Y, and Z. That is, the topic sets X and Z, and X, Y, and Z. The first consumption identifier corresponding to the target topic set is determined as the second consumption identifier. That is, the first consumption identifier V1 corresponding to the topic set X and Z, and the first consumption identifier V2 corresponding to the topic set X, Y, and Z are determined as the second consumption identifier.

[0101] For example, see Figure 7 , Figure 7 This is a schematic diagram of the second consumer identifier provided in the embodiments of this application, based on Figure 7 If the second topic is X and Z, then combine Figure 6 It can be determined that the target topic set including the second topic X and Z is topic set 1 and 3, and the first consumption identifier corresponding to topic set 1 is V2, and the first consumption identifier corresponding to topic set 3 is V1. Therefore, V1 and V2 are determined as the second consumption identifier.

[0102] Step 1022: Based on the first consumer identifier and the second consumer identifier, select the first consumer instance from the consumer group.

[0103] In practice, after determining the first consumer identifier and the second consumer identifier, the first consumer instance can be selected from the consumer group based on the first consumer identifier and the second consumer identifier. Specifically, the number of second consumer identifiers is at least one. The process of selecting the first consumer instance from the consumer group based on the first consumer identifier and the second consumer identifier can be as follows: for each consumer instance in the consumer group, the corresponding first consumer identifier is matched with each second consumer identifier to obtain a matching result; when the matching result indicates that there is a second consumer identifier that matches the first consumer identifier, the corresponding consumer instance is determined as the first consumer instance.

[0104] Following the example above, for consumer instances A, B, C, and D, the topics of messages that consumer instance A can consume are X, Y, and Z; the topics that consumer instance B can process are X and Y; the topics that consumer instance C can process are X and Z; and the topics that consumer instance D can process are also X and Y. Therefore, the first consumption identifier corresponding to consumer instance A is V2, the first consumption identifier corresponding to consumer instance C is V1, and the first consumption identifier corresponding to consumer instances D and B is V0. The second consumption identifiers are V1 and V2. The first consumption identifiers of consumer instances A, B, C, and D are matched with V1 and V2 respectively. Thus, V2 in the second consumption identifier matches the first consumption identifier of consumer instance A, and V1 in the second consumption identifier matches the first consumption identifier of consumer instance C. In other words, consumer instances A and C are the first consumer instances.

[0105] For example, see Figure 8 , Figure 8 This is a schematic diagram illustrating the process of filtering the first consumer instance provided in an embodiment of this application, based on... Figure 8 ,like Figure 8 As shown in a, Figure 8 There are four consumer instances. Consumer instance A has a first consumer identifier of V2, consumer instance C has a first consumer identifier of V1, and consumer instances D and B have first consumer identifiers of V0. If the second consumer identifiers are V1 and V2, then the first consumer identifiers of consumer instances A, B, C, and D are matched with V1 and V2 respectively. Thus, V2 in the second consumer identifiers matches the first consumer identifier of consumer instance A, and V1 in the second consumer identifiers matches the first consumer identifier of consumer instance C. That is, consumer instances A and C are the first consumer instances. Figure 8 As shown in b.

[0106] In some embodiments, in addition to determining the first consumer instance based on topic information, the administrator of the consumer group can also filter consumer instances by setting a disable flag. Specifically, as mentioned above, the topic information is obtained by parsing the consumption management information, that is, the topic information is included in the consumption management information. Thus, the process of determining the first consumer instance in the consumer group that needs to be allocated messages can also be that, when the consumption management information includes a disable flag, a second consumer instance that does not need to be allocated messages is filtered out from the consumer group based on the disable flag; wherein, the disable flag is the instance identifier of the consumer instance that does not need to be allocated messages; and the first consumer instance that needs to be allocated messages is determined based on the consumer instances in the consumer group other than the second consumer instance.

[0107] It should be noted that the disabled identifier can be, for example, the ID or IP of a consumer instance that does not need to be allocated messages; when the consumption management information includes topic information and a disabled identifier, and the topic information includes a second topic with a higher priority than the first topic, after filtering out the second consumer instance that does not need to be allocated messages from the consumer group, the process of determining the first consumer instance that needs to be allocated messages based on the consumer instances in the consumer group other than the second consumer instance is as follows: when the topic information includes a second topic with a higher priority than the first topic, the first consumer identifier of the consumer instances in the consumer group other than the second consumer instance is obtained, and the second consumer identifier of the second topic is obtained. Based on the first consumer identifier and the second consumer identifier, the first consumer instance is filtered out from the consumer instances in the consumer group other than the second consumer instance.

[0108] The process of obtaining the first consumer identifier of each consumer instance in the consumer group other than the second consumer instance, and obtaining the second consumer identifier of the second topic; and selecting the first consumer instance from the consumer instances in the consumer group other than the second consumer instance based on the first and second consumer identifiers, is similar to the process described above of obtaining the first consumer identifier of each consumer instance in the consumer group, obtaining the second consumer identifier of the consumer group, and selecting the first consumer instance from the consumer group based on the first and second consumer identifiers. Therefore, this embodiment will not be described in detail here.

[0109] It should be noted that if the topic information includes a second topic with higher priority than the first topic, the filtering process based on the first consumption identifier and the second consumption identifier can be executed first, followed by the filtering process based on the disabled identifier. Alternatively, the filtering process based on the disabled identifier can be executed first, followed by the filtering process based on the first consumption identifier and the second consumption identifier. The above process is described with the filtering process based on the disabled identifier executed first, followed by the filtering process based on the first consumption identifier and the second consumption identifier executed second. In practical applications, the execution order of the filtering process based on the first consumption identifier and the second consumption identifier and the filtering process based on the disabled identifier is not limited.

[0110] In practice, the process of selecting second consumer instances that do not need to be allocated messages from the consumer group based on the disable identifier can be as follows: obtain the instance identifier of each consumer instance in the consumer group; match the disable identifier with the instance identifier of each consumer instance to obtain the matching result; when the matching result indicates that the instance identifier matches the disable identifier, the consumer instance corresponding to the instance identifier is taken as the second consumer instance; thereby, the partitions that were previously allocated to the second consumer instance are unmounted.

[0111] It should be noted that unmounting partitions that were historically allocated to the second consumer instance means releasing the partition information that was historically allocated to the second consumer instance, so that the second consumer instance is no longer recognized and used by the operating system. For example, for consumer instances A, B, C, and D, if the ID or IP of consumer instance D is disabled, then consumer instance D is selected from these four consumer instances, and based on the remaining consumer instances A, B, and C, the first consumer instance that needs to be allocated messages is determined.

[0112] For example, see Figure 9 , Figure 9 This is a schematic diagram illustrating the process of filtering the first consumer instance provided in an embodiment of this application, based on... Figure 9 ,like Figure 9 As shown in a, Figure 9 There are four consumer instances. If the ID or IP of consumer instance D is disabled, then consumer instance D is selected from these four consumer instances. Therefore, consumer instances A, B, and C are determined as the first consumer instances to be allocated messages. Figure 9 As shown in b.

[0113] In other embodiments, after obtaining the topic information corresponding to the consumer group, if the topic information only includes the first topic, the consumption information of each consumer instance in the consumer group is directly obtained, and the message of the first topic is allocated to each consumer instance based on the consumption information.

[0114] It should be noted that consumption information is used to indicate the consumption capacity of the corresponding consumer instance, that is, the topic to be consumed. Based on the consumption information, messages of the first topic are allocated to each consumer instance. At the same time, if the topic information only includes the first topic, and the first topic is the topic subscribed to by the consumer instances in the consumer group, that is, the first topic is determined when the consumer instance registers, then there will be no situation where each consumer instance in the consumer group cannot process messages of the first topic. In other words, each consumer instance in the consumer group can consume messages of the first topic. Therefore, the consumption information of at least one consumer instance in the consumer group can be directly obtained, and messages of the first topic can be allocated to each consumer instance based on the consumption information.

[0115] In practice, if the topic information only includes the first topic, the consumer instances can be filtered based on their load. Specifically, when the topic information only includes the first topic, the load information of each consumer instance in the consumer group is obtained; based on the load information, the fifth consumer instance with a load less than or equal to the load threshold is filtered out from the consumer group; based on the consumption information of the fifth consumer instance, the message of the first topic is assigned to the fifth consumer instance.

[0116] It's important to note that the load information here refers to the load of consumer instances. The load threshold is preset and not limited here. Based on the load information, after selecting the fifth consumer instance from the consumer group whose load is less than or equal to the load threshold, the consumption information of each fifth consumer instance is obtained. Based on this consumption information, the messages of the first topic are assigned to the fifth consumer instance. In this way, consumer instances are filtered based on their load status, ensuring that only unloaded consumer instances are selected to process messages. This guarantees the stability of consumer instances when consuming messages and reduces the probability of problems.

[0117] Furthermore, if the topic information includes a second topic with a higher priority than the first topic and a disabled identifier is obtained, after completing the filtering process based on the first consumer identifier and the second consumer identifier and the filtering process based on the disabled identifier, the consumer instances can be further filtered based on the load of the consumer instances. Here, the process of filtering consumer instances based on the load of the consumer instances is as described above, and will not be repeated in this embodiment.

[0118] It should be noted that the foregoing descriptions refer to obtaining topic information including a first topic, obtaining topic information including both a first topic and a second topic, or obtaining topic information and a disabling identifier. In some embodiments, only the disabling identifier may be obtained. Here, if only the disabling identifier is obtained, the topic information does not include the second topic, but only includes the first topic determined by the consumer instance during registration. Therefore, the message processing based on the disabling identifier may involve obtaining the disabling identifier corresponding to the consumer group, and based on the disabling identifier, filtering out consumer instances from the consumer group that do not need to be allocated messages; designating the consumer instances other than those that do not need to be allocated messages as other consumer instances, and obtaining the consumption information of the other consumer instances; and based on the consumption information of the other consumer instances, allocating the topic message to the other consumer instances.

[0119] The process of filtering out consumer instances that do not need to be allocated messages from the consumer group based on the disabled identifier is similar to the process of filtering out second consumer instances that do not need to be allocated messages from the consumer group based on the disabled identifier described above, and will not be repeated here in the embodiments of this application.

[0120] Meanwhile, as mentioned above, topic information and disabling identifiers can exist in message management information. Here, if the topic information only includes the first topic, then the topic information may not be obtained from the message management information, but rather determined by the consumer instance during registration. If the topic information includes both the first and second topics, then the second topic is determined based on the message management information, meaning the topic information is obtained from the message management information. Based on this, the above-mentioned obtaining only the disabling identifier means obtaining only the disabling identifier from the message management information, but not obtaining the topic information including the second topic from the message management information. In other words, the topic information is determined by the consumer instance during registration, and here the topic information only includes the first topic. Based on the consumption information of other consumer instances, the messages of the topic are allocated to other consumer instances, that is, based on the consumption information of other consumer instances, the messages of the first topic are allocated to other consumer instances.

[0121] Step 103: Obtain the consumption information of the first consumer instance. The consumption information is used to indicate the subject of the message that the first consumer instance needs to process.

[0122] It should be noted that consumption information is used to indicate the spending power of the corresponding consumer instance, that is, the topic to be consumed. For example, see [link to relevant documentation]. Figure 8 ,based on Figure 8 In the context of b, the message topics that the first consumer instance needs to process are X and Z.

[0123] Step 104: Based on the consumption information of the first consumer instance, assign the message of the second topic to the first consumer instance.

[0124] In practice, the process of allocating messages of the second topic to the first consumer instance based on the consumption information of the first consumer instance can be as follows: based on the consumption information of the first consumer instance, determine the partition corresponding to the first consumer instance, and allocate messages of the second topic to the first consumer instance based on the partition corresponding to the first consumer instance.

[0125] Specifically, the process of determining the partition corresponding to the first consumer instance based on the consumption information of the first consumer instance involves first determining the topic that the first consumer instance needs to process, i.e., the second topic, based on the consumption information of the first consumer instance; obtaining the number of partitions of the second topic, and dividing the second topic into at least two partitions based on the number of partitions; and assigning the at least two partitions to the first consumer instance to obtain the partition corresponding to the first consumer instance. Here, if there are multiple first consumer instances, the partitions corresponding to different first consumer instances will be different.

[0126] For example, taking the second topic as X and Z, if the number of partitions is three, topic X is divided into three partitions: XP-1, XP-2, and XP-3, and topic Z is divided into three partitions: ZP-1, ZP-2, and ZP-3. If the first consumer instance is consumer instance A and consumer instance C, then consumer instance A is allocated three partitions: XP-1, XP-3, and ZP-2, and consumer instance C is allocated three partitions: XP-2, ZP-1, and ZP-3.

[0127] Then, for the partition corresponding to the first consumer instance, the process of allocating the messages of the second topic to the first consumer instance specifically involves determining the messages in the partition corresponding to each first consumer instance and then allocating the messages in each partition to the corresponding first consumer instance in sequence. Continuing with the example above, if consumer instance A is allocated partitions XP-1, XP-3, and ZP-2, and consumer instance C is allocated partitions XP-2, ZP-1, and ZP-3, then first determine the messages in each partition, then allocate the messages in the XP-1, XP-3, and ZP-2 partitions to consumer instance A in sequence, and allocate the messages in the XP-2, ZP-1, and ZP-3 partitions to consumer instance C in sequence.

[0128] In practice, the partition corresponding to the first consumer instance, i.e., the partition information of the first consumer instance, is determined when the first consumer instance is registered. In some embodiments, the administrator of the consumer group can also control the messages processed by the consumer instance by setting the partition information. Specifically, as mentioned above, the topic information is obtained by parsing the consumption management information, i.e., the topic information is included in the consumption management information. Therefore, the process of allocating the message of the second topic to the first consumer instance based on the consumption information of the first consumer instance can be that when the consumption management information includes partition information, the partition information is used to allocate the message of the second topic to the first consumer instance; wherein, the partition information is used to indicate the partition allocated to each first consumer instance.

[0129] It should be noted that the partition information indicates the partition allocated to each first consumer instance, that is, the correspondence between the first consumer instance and the specific partition. Therefore, the process of using partition information to allocate messages of the second topic to the first consumer instance can be as follows: based on the partition message, determine the partition corresponding to the first consumer instance, then determine the messages in the partition corresponding to each first consumer instance, and then allocate the messages in each partition to the corresponding first consumer instance in sequence. For example, in the above example, when the partition message indicates that the partitions corresponding to consumer instance A are XP-1, XP-3, and ZP-2, and the partitions corresponding to consumer instance C are XP-2, ZP-1, and ZP-3, then first determine the messages in each partition, then allocate the messages of the three partitions XP-1, XP-3, and ZP-2 to consumer instance A in sequence, and allocate the messages of the three partitions XP-2, ZP-1, and ZP-3 to consumer instance C in sequence.

[0130] It should be noted that if the topic information includes both a first topic and a second topic, then the number of partitions of the second topic is obtained, and based on the number of partitions, the second topic is divided into at least two partitions; then, the at least two partitions are assigned to the first consumer instance respectively to obtain the partitions corresponding to the first consumer instance; then, the messages in the partitions corresponding to each first consumer instance are determined, and the messages in each partition are assigned to the corresponding first consumer instance in sequence.

[0131] If the topic information only includes the first topic, then obtain the number of partitions for the first topic, and divide the first topic into at least two partitions based on the number of partitions; then, assign the at least two partitions to each consumer instance in the consumer group to obtain the partitions corresponding to each consumer instance; then determine the messages in the partitions corresponding to each consumer instance, and assign the messages in each partition to the corresponding consumer instance in sequence; wherein, if the consumer instances are filtered based on the load of the consumer instances, after dividing the first topic into at least two partitions, assign the at least two partitions to the fifth consumer instance to obtain the partitions corresponding to each fifth consumer instance; then determine the messages in the partitions corresponding to each fifth consumer instance, and assign the messages in each partition to the corresponding fifth consumer instance in sequence.

[0132] In this way, the administrator of the consumer group can specify the partitions corresponding to the consumer instances by setting partition information, so that the consumer instances can process messages in specific partitions, thereby strengthening the management of the consumer group.

[0133] It should be noted that the processes described above are respectively the process of obtaining topic information including a first topic, obtaining topic information including a first topic and a second topic, or obtaining topic information and partition information. In some embodiments, only partition information may be obtained. Here, if only partition information is obtained, the topic information does not include the second topic, but only includes the first topic determined by the consumer instance during registration. Based on this, obtaining only partition information means obtaining only partition information from the message management information, but not obtaining topic information including the second topic from the message management information. That is, the topic information is determined by the consumer instance during registration, and the topic information here only includes the first topic.

[0134] If the partition information exists in the message management information, then the partition information is set by the consumer group administrator and is different from the partition information when the consumer instance is registered. If the partition information does not exist in the message management information, then the partition information is determined when the consumer instance is registered.

[0135] Accordingly, if the topic information exists in the message management information, the topic in the topic information includes the topic specified by the consumer group's administrator. If the topic information does not exist in the message management information, the topic in the topic information is the topic determined when the consumer instance is registered. For example, the topic subscribed to when the consumer instance is registered can be the first topic mentioned above, and the topic specified by the consumer group's administrator can be the second topic mentioned above. Since the second topic is specified, the second topic has a higher priority than the first topic. The topic specified by the consumer group's administrator can be the same as the topic determined when the consumer instance is registered. This application embodiment does not limit this. If they are the same, it can be considered that the topic information only includes the first topic.

[0136] It should be noted that the topic information, disabled labels, and partition information mentioned above can all be set by the administrator of the consumer group, that is, determined through the consumer management information. The administrator of the consumer group can set at least one of the topic information, disabled labels, and partition information in the consumer management information.

[0137] In some embodiments, the situation is as follows: the consumer group administrator sets one of the following in the consumption management information: topic information, disable flag, and partition information. If the consumer group administrator only sets the topic information, it means that the consumer group administrator has specified a topic. Therefore, it is necessary to determine whether there is a second topic in the specified topic that is different from the first topic when the consumer instance is registered. Therefore, when the topic information includes both the first topic and the second topic, it is necessary to filter the consumer instances that can handle the second topic, that is, to determine the first consumer instance in the consumer group that needs to be allocated messages. The consumption information of the first consumer instance is obtained, and based on the consumption information of the first consumer instance, the message of the second topic is allocated to the first consumer instance.

[0138] When the topic information only includes the first topic, the consumption information of each consumer instance in the consumer group is directly obtained, and the message of the first topic is distributed to each consumer instance based on the consumption information of each consumer instance.

[0139] Here, "assigning messages of the second topic to the first consumer instance or assigning messages of the first topic to each consumer instance" means assigning messages of the second topic to the first consumer instance or assigning messages of the first topic to each consumer instance according to the partition information determined when the consumer instance is registered.

[0140] If the consumer group administrator only sets a disable flag, it means that the consumer group administrator has not specified a topic. The topic information is determined when the consumer instance is registered. Therefore, the following steps are taken: obtain the topic information of the first topic and the disable flag corresponding to the consumer group; based on the disable flag, filter out consumer instances in the consumer group that do not need to be allocated messages; treat the consumer instances in the consumer group that do not need to be allocated messages as other consumer instances and obtain their consumption information; based on the consumption information of the other consumer instances, allocate the messages of the first topic to the other consumer instances.

[0141] In this context, allocating messages for the first topic to other consumer instances means distributing messages for the first topic to other consumer instances according to the partition information determined when the consumer instance is registered.

[0142] If the consumer group manager only sets partition information, it means that the consumer group manager has not specified a topic. The topic information is determined when the consumer instance is registered. Therefore, the topic information including the first topic and the partition information corresponding to the consumer group are obtained. Using the partition information, the messages of the first topic are distributed to at least one consumer instance included in the consumer group.

[0143] In some embodiments, the scenario involves the consumer group administrator setting two of the following in the consumption management information: topic information, disable flag, and partition information. If the consumer group administrator only sets the topic information and disable flag, then when the topic information includes both a first topic and a second topic, the message processing procedure based on the topic information and disable flag is as described above and will not be repeated here. When the topic information only includes the first topic, then based on the disable flag, consumer instances that do not need to be allocated messages are selected from the consumer group. Consumer instances other than those that do not need to be allocated messages are designated as other consumer instances, and their consumption information is obtained. Based on the consumption information of the other consumer instances, the message for the first topic is allocated to the other consumer instances.

[0144] Specifically, when the topic information includes a first topic and a second topic, allocating messages of the second topic to the first consumer instance means allocating messages of the second topic to the first consumer instance according to the partition information determined when the consumer instance is registered; when the topic information includes the first topic, allocating messages of the first topic to other consumer instances based on the consumption information of other consumer instances means allocating messages of the first topic to other consumer instances according to the partition information determined when the consumer instance is registered.

[0145] If the consumer group administrator only sets topic information and partition information, then when the topic information includes a first topic and a second topic, the message processing process based on the topic information and partition messages is as described above and will not be repeated here; when the topic information includes the first topic, the messages of the first topic are distributed to each consumer instance in the consumer group according to the set partition information.

[0146] If the consumer group administrator only sets partition information and a disable flag, then the topic information, including the first topic, is obtained; based on the disable flag, consumer instances that do not need to be assigned messages are selected from the consumer group; the consumer instances other than those that do not need to be assigned messages are treated as other consumer instances, and the messages of the first topic are assigned to the other consumer instances according to the set partition information.

[0147] It should be noted that if the consumer group administrator does not set topic information, message processing will be performed according to the topic information when the consumer instance is registered. That is, only the topic information of the first topic will be included, and the first consumer instance will not be selected. If the consumer group administrator does not set a disable flag, consumer instances will not be selected based on the disable flag. If the consumer group administrator does not set partition information, the partition information when the consumer instance is registered will be determined, and messages will be distributed to the consumer instance according to the partition information.

[0148] In other embodiments, the scenario involves a consumer group administrator setting topic information, a disable flag, and partition information in the consumption management information. Here, the consumption management information may include topic information, a disable flag, and partition information. As mentioned earlier, the disable flag is the instance identifier of a consumer instance that does not need to be allocated messages, and the partition information indicates the partition allocated to each consumer instance. Thus, the topic information corresponding to the consumer group is obtained, wherein the consumer group includes at least one consumer instance that handles messages. When the topic information includes a second topic with a higher priority than the first topic, a third consumer instance that cannot process messages is selected from the consumer group based on the disable flag. The consumer instances other than the third consumer instance among the at least one consumer instance are designated as fourth consumer instances. A first consumption flag corresponding to each fourth consumer instance is obtained, and a second consumption flag for the consumer group is obtained. A first consumer instance is selected from the fourth consumer instances based on the first and second consumption flags. Then, using the partition information, messages from the second topic are allocated to the first consumer instance.

[0149] It should be noted that since the topic information is included in the message management information, the topic information includes the second topic. The execution order of the filtering process based on the disable identifier and the filtering process based on the first consumption identifier and the second consumption identifier can be either first, followed by the filtering process based on the first consumption identifier and the second consumption identifier, or vice versa. The above description assumes that the filtering process based on the disable identifier is executed first, followed by the filtering process based on the first consumption identifier and the second consumption identifier. In practical applications, the execution order of the filtering process based on the first consumption identifier and the second consumption identifier and the filtering process based on the disable identifier is not limited.

[0150] Meanwhile, the process of filtering out a third consumer instance that cannot process messages from the consumer group based on a disabled identifier is similar to the process of filtering out a second consumer instance that does not need to be allocated messages from the consumer group based on a disabled identifier, as described above. Obtaining the second consumer identifier of the consumer group and filtering out a first consumer instance from the fourth consumer instance based on the first and second consumer identifiers is similar to the process of obtaining the second consumer identifier of the consumer group and filtering out a first consumer instance from the consumer group based on the first and second consumer identifiers, as described above. As for using partition information to allocate messages of the second topic to the first consumer instance, it is also as described above, and will not be repeated here in the embodiments of this application.

[0151] In some embodiments, after allocating messages of the second topic to the first consumer instance based on the consumption information of the first consumer instance, the instance information of each consumer instance in the consumer group can be obtained; wherein, the instance information includes at least one of the following: the instance identifier of the consumer instance, the group identifier of the consumer group, the topic subscribed to by the consumer instance, status information indicating whether the consumer instance is consuming normally, and partition information; and a consumption report is generated based on the instance information of each consumer; wherein, the consumption report is used to view the consumption status of each consumer instance.

[0152] It should be noted that, regarding the topics subscribed to by a consumer instance, when the topic information only includes the first topic, the topic subscribed to by the consumer instance is also the first topic; when the topic information includes both the first and second topics, the topic subscribed to by the consumer instance is also the second topic. As for the status information used to indicate whether a consumer instance is consuming normally, as mentioned above, consumer instances can be filtered based on a disabled flag or a consumption flag. Some consumer instances cannot consume, so the status information can be used to indicate the current status of a consumer instance, such as whether it is working normally, or whether it is a mismatched node (e.g., a consumer instance filtered out based on a consumption flag), or whether it is an abnormal node (e.g., a consumer instance filtered out based on a disabled flag).

[0153] For example, see Figure 10 , Figure 10 This is a schematic diagram of a consumption report provided in an embodiment of this application, based on Figure 10 The first column is the instance identifier of the consumer instance, the second column is the group name of the consumer group, which is also the group identifier of the consumer group, the third column is the topic subscribed to by the consumer instance, where set-1 is the first topic, set-2 is the second topic, the third column is the status information used to indicate whether the consumer instance is consuming normally, and the fourth column is the partition information.

[0154] Thus, based on consumption reports, managers of consumer groups can view the consumption status of each consumer instance, which facilitates the management of consumer groups and improves the convenience and efficiency of managing consumers.

[0155] In some embodiments, there are multiple consumer groups. Before obtaining the topic information corresponding to a consumer group, the multiple consumer groups can be sorted to obtain a consumer group sequence. The process of obtaining the topic information corresponding to a consumer group can be based on the consumer group sequence, determining when the processing time point of the consumer group arrives, and then obtaining the topic information corresponding to the consumer group.

[0156] It should be noted that the process of sorting multiple consumer groups to obtain a consumer group sequence can be based on the load of the consumer groups, for example, starting from the consumer group with the smallest load and sorting multiple consumer groups sequentially based on the load size; or it can be based on the priority of the consumer groups, for example, obtaining the priority of each consumer group and sorting multiple consumer groups sequentially based on the priority of the consumer group with the highest priority; or it can be based on the consumption capacity of the consumer groups, etc. There are many ways to sort multiple consumer groups, and this application embodiment does not limit them.

[0157] Applying the above embodiments of this application, when the obtained topic information indicates that a second topic with a higher priority than the first topic exists in the consumer group, the first consumer instance in the consumer group that needs to be allocated a message partition is determined. Then, the consumption information of the first consumer instance is obtained, and based on the consumption information of the first consumer instance, the messages of the second topic are allocated to the first consumer instance. Thus, after obtaining the topic information for managing the consumer group, a centralized management model is adopted to manage consumers in the consumer group based on the topic information. Compared with the decentralized management model in related technologies, this simplifies the management process and improves the convenience and efficiency of managing consumers.

[0158] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0159] In related technologies, Consumer Group is the data consumption management mechanism adopted by mainstream message queues (including Kafka, Pulsar, etc.). There are multiple consumers or consumer instances in a consumer group, which share a common ID, called the consumer group ID or group name. All consumers in the consumer group are coordinated by the message queue server to consume all partitions of the subscribed topics.

[0160] When a consumer (consumer instance) starts up, it configures the name of its consumer group and the set of topics it subscribes to. After receiving the registration request of the consumer instance, the MQ server identifies the consumer group to which the consumer belongs based on the name of the consumer group it carries. Consumers are grouped into the same consumer group for management according to the name of the consumer group. At the same time, the MQ server allocates the topics to consumers subscribing to the topics according to the allocation strategy based on the set of topics subscribed to by each consumer in the consumer group (first topic or second topic).

[0161] For example, three topics are allocated in the message queue (MQ): X, Y, and Z. Each topic is configured with three partitions: P-1, P-2, and P-3. When the three consumers start, they are set to the consumer group name Demo-1 (i.e., consumer group-1). If all consumers in the consumer group subscribe to the same topic, see [link to relevant documentation]. Figure 11 , Figure 11 This is a schematic diagram illustrating consumer-related information in the related technologies provided in the embodiments of this application, based on... Figure 11 This consumer group will form a consumption allocation relationship as shown in Figure 11 according to the allocation strategy; at the same time, three other consumers set their consumer group name to Demo-2 (i.e., consumer group-2) when they start up. If consumers within the consumer group subscribe to different topics during startup or operation, please refer to [the relevant documentation]. Figure 12 , Figure 12 This is a schematic diagram illustrating consumer-related information in the related technologies provided in the embodiments of this application, based on... Figure 12 This consumer group will form, according to the allocation strategy, as follows: Figure 12 The consumption distribution relationship is shown.

[0162] against Figure 11 as well as Figure 12 The consumer group shown adopts a decentralized consumer group model. Therefore, the set of topics subscribed to by the consumer group is composed of the set of topics independently submitted by each consumer in the group, thus forming the consumer group's subscribed topic set. When managing consumers in the consumer group, each consumer must make adjustments individually for the changes to take effect. This can easily lead to problems where different consumers consume different data due to inconsistencies in the topic sets they submit. At the same time, it does not consider the situation where the message queue service provider and the user belong to different technical groups. Therefore, it does not provide an effective way for the consumer group manager to manage the consumers in the consumer group themselves.

[0163] Therefore, the following three problems exist in the consumer group management solution: 1. Regardless of whether server-side or client-side partition allocation management is used, adjusting the topic set subscribed to by a consumer group must be achieved by adjusting the topic set subscribed to by each consumer instance within that consumer group; 2. When a consumer in a consumer group experiences a consumption anomaly but remains alive, the allocated partition will experience consumption lag. Only by stopping the process of the consumer, causing it to exit the consumer group, can the impact of the abnormal consumer on the consumer group's consumption be eliminated; 3. When adjusting the partition allocation strategy of a consumer group, for consumer groups using client-side partition allocation management, the client implementation or configuration of all consumers in the consumer group needs to be adjusted; for consumer groups using server-side partition allocation management, a single global strategy is used and cannot be adjusted according to business needs.

[0164] Based on this, this application provides a message processing method that proposes a centralized management scheme for consumer groups. The scheme allows the consumer group owner, i.e., the consumer group manager, to manage (including view and change) the set of topics subscribed to by the group. The consumer group owner also controls the partition allocation strategy to decide whether to allow consumers within the group to join the consumption. This solves the aforementioned problems encountered when using a decentralized consumer group model to manage consumers.

[0165] In actual implementation, the entire technical solution is divided into the following parts:

[0166] First, the message processing capabilities of consumers are identified by strings. Specifically, which topics a consumer consumes and how it consumes and processes messages are client-side business behaviors. Often, topics and consumer processing logic are bound together. For both the consumer implementation and the data processing side, this capability is predictable and identifiable. The ability of a consumer to consume specific topics and process message formats needs to be abstracted and identified through field information (consumer identifier) ​​to represent the message processing capabilities of different consumers. Figure 6 As shown.

[0167] Second, the ability to differentiate the subscription topic set specified by the consumer group is added. Specifically, the subscription topic set registered by each consumer in the consumer group is called Set-1 (the first topic, i.e., Set-1). On the server side, a topic set that the consumer group must subscribe to is added by API, tools, configuration files, etc., called Set-2 (the second topic, i.e., Set-2). The execution priority of Set-2 is higher than that of Set-1. When only Set-1 exists, the system performs topic subscription and data consumption processing according to Set-1. When Set-2 is configured, the system uses Set-2 as the subscription topic set of the consumer group and performs the corresponding data consumption processing.

[0168] Third, when a consumer starts up, in addition to filling in its consumer group and the set of topics it has subscribed to, it also needs to report field information indicating its own message processing capabilities. Then, the set of topics filled in during consumer registration is defined as the initial topic set Set-1, such as... Figure 4 As shown.

[0169] Fourth, the ability to specify the subscription topic set for a consumer group has been added. Specifically, the consumer group owner can specify the subscription topic set Set-2 for the consumer group through the management interface. The topic set Set-2 specified through the interface has higher execution priority than the topic set Set-1 filled in during consumer registration. After specifying the topic set Set-2, the field information corresponding to the topic set Set-2 is determined, such as... Figure 7 As shown;

[0170] Fifth, the management node adds a consumer group control interface, which allows the owner of the consumer group to manage the consumption of the consumer group, including changing the set of subscribed topics, the field information of allowed consumption for each topic, and the Consumer ID or IP of prohibited consumption in the consumer group;

[0171] Sixth, before performing consumer load balancing, the management node checks whether the consumer group has consumer management information specified by the consumer group administrator. If not, it allocates data to different partitions according to the allocation scheme when the consumer was registered. If the consumer group has consumer management information specified by the consumer group administrator, the following process is executed:

[0172] First, based on the prohibited consumer ID in the consumption management information, exclude consumers from the consumption group who are prohibited from making purchases, and release the consumption partition held by that consumer. For example, when the prohibited consumer ID is... Figure 9 When consumer D is in consumer group a, exclude consumer D from the consumer group and release the partition information related to topics X and Y previously assigned to consumer D, resulting in the following: Figure 9 Consumer b is shown.

[0173] Then, for consumers whose field information is outside the allowed range of Set-2, an instruction is issued to release the consumer's held consumer partition, for example, such as... Figure 8 As shown, if the consumer's field information is V1 and V2 within the allowed range of Set-2, then consumers whose field information is not V1 and V2 are excluded, that is, consumers B and D are excluded, and the partition information related to topic X previously allocated to consumers B and D is released, resulting in the following... Figure 8 Consumer b is shown.

[0174] Next, consumers who meet the conditions in the consumer group are partitioned and allocated. Specifically, the partition information related to topic X previously assigned to consumers A and C is updated to the partition information related to topics X and Z. Then, the management node performs partition load balancing based on the partition information of consumers A and C.

[0175] Finally, as Figure 10 As shown, the management node archives the partition allocation results, allowing managers of consumer groups to view the consumption status of each consumer.

[0176] It should be noted that, see Figure 13 , Figure 13 This is a schematic diagram illustrating the process of managing consumer groups provided in an embodiment of this application, based on... Figure 13 In addition to managing the collection of topics subscribed to by consumer groups and the consumption of consumers in consumer groups, this application can also specify the method for allocating consumer groups to different regions and define consumption strategies. Compared with the past, where decentralized consumers specified the scheme one by one, this application is simpler and more efficient.

[0177] Thus, this application allows for centralized adjustment of the topic set subscribed by a consumer group, rather than individual consumers within the group adjusting it. This enables rapid and continuous operation of the consumption process. Furthermore, the consumer group manager oversees the consumers, allowing the business to manage consumption autonomously without the need for message queue maintenance personnel. Simultaneously, partition allocation algorithms can be specified according to business needs, and adjustments only require the server-side to take effect globally. Moreover, while mainstream message queues currently employ decentralized management for consumer groups, the dispersed nature of participants leads to management difficulties and delayed policy implementation. This application, by adding centralized management on top of decentralized management, improves the convenience and efficiency of consumer management.

[0178] Applying the above embodiments of this application, when the obtained topic information indicates that a second topic with a higher priority than the first topic exists in the consumer group, the first consumer instance in the consumer group that needs to be allocated a message partition is determined. Then, the consumption information of the first consumer instance is obtained, and based on the consumption information of the first consumer instance, the messages of the second topic are allocated to the first consumer instance. Thus, after obtaining the topic information for managing the consumer group, a centralized management model is adopted to manage consumers in the consumer group based on the topic information. Compared with the decentralized management model in related technologies, this simplifies the management process and improves the convenience and efficiency of managing consumers.

[0179] The following description continues to illustrate the exemplary structure of the message processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the message processing device 455 of the memory 450 may include:

[0180] The first acquisition module 4551 is used to acquire topic information corresponding to a consumer group, wherein the consumer group includes at least one consumer instance of the message;

[0181] The determining module 4552 is used to determine the first target consumer instance in the consumer group that needs to be allocated a message partition when the topic information indicates that there is a second topic in the consumer group with a higher priority than the first topic, wherein the first topic is the message topic subscribed to by the consumer instance in the consumer group;

[0182] The second acquisition module 4553 is used to acquire the consumption information of the first target consumer instance, wherein the consumption information is used to indicate the topic of the message that the first target consumer instance needs to process.

[0183] The allocation module 4554 is used to allocate messages of the second topic to the first target consumer instance based on the consumption information of the first target consumer instance.

[0184] In some embodiments, the determining module 4552 is further configured to obtain a first consumption identifier for each consumer instance in the consumer group and obtain a second consumption identifier for the consumer group; the first consumption identifier is used to indicate the topic of a message that the consumer instance can consume, and the second consumption identifier is used to indicate the first consumption identifier of a second consumer instance, wherein the second consumer instance is a consumer instance in the consumer group that can consume messages on the second topic; and the first consumer instance is selected from the consumer group based on the first consumption identifier and the second consumption identifier.

[0185] In some embodiments, the determining module 4552 is further configured to obtain a mapping relationship between the first consumption identifier and the topic set; based on the mapping relationship, determine a topic set corresponding to different first consumption identifiers, wherein the topic set includes at least one topic; determine a target topic set including the second topic from each topic set corresponding to the first consumption identifier, and determine the first consumption identifier corresponding to the target topic set as the second consumption identifier.

[0186] In some embodiments, the determining module 4552 is further configured to analyze each consumer instance to obtain the topics of messages that the consumer instance can consume; cluster the topics of messages that each consumer instance can consume to obtain at least one cluster, and the topics in each cluster constitute a topic set; generate a cluster identifier that corresponds one-to-one with the cluster, use the cluster identifier as the first consumption identifier, and establish a mapping relationship between the first consumption identifier and the topic set.

[0187] In some embodiments, the number of the second consumer identifiers is at least one, and the determining module 4552 is further configured to match the corresponding first consumer identifier with each second consumer identifier for each consumer instance in the consumer group to obtain a matching result; when the matching result indicates that there is a second consumer identifier that matches the first consumer identifier, the corresponding consumer instance is determined as the first consumer instance.

[0188] In some embodiments, the topic information is included in the consumption management information. The determining module 4552 is further configured to, when the consumption management information includes a disable identifier, filter out a second consumer instance that does not need to be allocated a message from the consumer group based on the disable identifier; wherein, the disable identifier is the instance identifier of the consumer instance that does not need to be allocated a message; and determine the first consumer instance that needs to be allocated a message based on the consumer instances in the consumer group other than the second consumer instance.

[0189] In some embodiments, the determining module 4552 is further configured to obtain the instance identifier of each consumer instance in the consumer group; match the disabled identifier with the instance identifier of each consumer instance to obtain a matching result; when the matching result indicates that the instance identifier matches the disabled identifier, the consumer instance corresponding to the instance identifier is taken as the second consumer instance; and unmount the partition historically allocated to the second consumer instance.

[0190] In some embodiments, the topic information is included in the consumption management information, and the allocation module 4554 is further configured to use the partition information to allocate the message of the second topic to the first consumer instance when the consumption management information includes partition information; wherein the partition information is used to indicate the partition allocated to each of the first consumer instances.

[0191] In some embodiments, the apparatus further includes a third acquisition module, which is configured to acquire instance information of each consumer instance in the consumer group; wherein the instance information includes at least one of the following: an instance identifier of the consumer instance, a group identifier of the consumer group, a topic subscribed to by the consumer instance, status information indicating whether the consumer instance is consuming normally, and partition information; and generate a consumption report based on the instance information of each consumer; wherein the consumption report is used to view the consumption status of each consumer instance.

[0192] In some embodiments, the first acquisition module 4551 is further configured to, in response to receiving consumption management information, parse the consumption management information to obtain topic information corresponding to the consumer group, wherein the consumption management information is configured by the manager of the consumer group; or, obtain the pre-set topic information from the storage structure.

[0193] In some embodiments, the consumption management information further includes a disable identifier and partition information. The disable identifier is an instance identifier of a consumer instance that does not need to be allocated messages, and the partition information is used to indicate the partition allocated to each consumer instance. The determining module 4552 is further configured to, based on the disable identifier, filter out a third consumer instance from the consumer group that cannot process messages; designate the consumer instances other than the third consumer instance among the at least one consumer instance as fourth consumer instances; obtain a first consumption identifier corresponding to each fourth consumer instance, and obtain a second consumption identifier of the consumer group; filter out the first consumer instance from the fourth consumer instances based on the first consumption identifier and the second consumption identifier; and the allocation module 4554 is further configured to, using the partition information, allocate messages of the second topic to the first consumer instance.

[0194] In some embodiments, the apparatus further includes a fourth acquisition module, which is configured to acquire load information of each consumer instance in the consumer group when the topic information includes only the first topic; based on the load information, select a fifth consumer instance from the consumer group whose load is less than or equal to a load threshold; and based on the consumption information of the fifth consumer instance, allocate the message of the first topic to the fifth consumer instance.

[0195] In some embodiments, the number of consumer groups is multiple, and the device further includes a sorting module, which is used to sort the multiple consumer groups to obtain a consumer group sequence; the first acquisition module 4551 is also used to determine, based on the consumer group sequence, when the processing time point of the consumer group arrives, to acquire the topic information corresponding to the consumer group.

[0196] This application provides a computer program product including computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the message processing method described in this application.

[0197] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the message processing method provided in this application, for example, such as... Figure 3 The message processing method is shown.

[0198] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or CD-ROM, etc.; or it may be a device that includes one or any combination of the above-mentioned memories.

[0199] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0200] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0201] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0202] In summary, the embodiments of this application have the following beneficial effects:

[0203] (1) After obtaining the topic information for managing the consumer group, a centralized management model is adopted to manage the consumers in the consumer group based on the topic information. Compared with the decentralized management model in related technologies, the management process is simplified and the convenience and efficiency of managing consumers are improved.

[0204] (2) Consumer instances are filtered based on their load status, thus selecting those that are not overloaded to process messages. This ensures the stability of consumer instances when consuming messages and reduces the probability of problems.

[0205] (3) The administrator of the consumer group can specify the partitions corresponding to the consumer instances by setting partition information, so that the consumer instances can process messages in specific partitions, thereby strengthening the management of the consumer group.

[0206] (4) Based on the consumption report, the manager of the consumer group can view the consumption of each consumer instance, which makes it easier for the manager of the consumer group to manage the consumer group and improves the convenience and efficiency of managing consumers.

[0207] (5) This application allows for centralized adjustment of the topic set subscribed by the consumer group, rather than adjustment by individual consumers in the consumer group. This enables fast and continuous operation of the consumption business. Moreover, the consumer group manager manages the consumers in the consumer group, allowing the business to manage consumption itself without the need for message queue maintenance personnel. At the same time, the partition allocation algorithm can be specified according to business needs, and only the server needs to be adjusted for global effect. In addition, the current mainstream MQ uses a decentralized management method for the management of consumer groups. However, since the participants in the consumer group are in a discrete state, the management of the consumer group has drawbacks such as management difficulties and untimely policy effects. This application adds centralized management on the basis of supporting decentralized management, which can improve the convenience and efficiency of managing consumers.

[0208] It should be noted that in this application embodiment, data related to obtaining consumer identifiers, theme information, partition information, and disabling identifiers are involved. When this application embodiment is applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0209] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A message processing method, characterized in that, The method includes: Obtain the topic information corresponding to the consumer group, wherein the consumer group includes at least one consumer instance of the message; When the topic information includes a second topic with a higher priority than the first topic, the first consumer instance in the consumer group that needs to be allocated messages is determined, wherein the first topic is the topic subscribed to by the consumer instance in the consumer group; Obtain the consumption information of the first consumer instance, wherein the consumption information is used to indicate the subject of the message that the first consumer instance needs to process; Based on the consumption information of the first consumer instance, the message of the second topic is assigned to the first consumer instance.

2. The method as described in claim 1, characterized in that, Determining the first consumer instance in the consumer group that needs to be allocated messages includes: Obtain the first consumer identifier for each consumer instance in the consumer group, and obtain the second consumer identifier for the consumer group; The first consumption identifier is used to indicate the topic of the message that the consumer instance can consume, and the second consumption identifier is used to indicate the first consumption identifier of the second consumer instance, wherein the second consumer instance is a consumer instance in the consumer group that can consume messages on the second topic; Based on the first consumer identifier and the second consumer identifier, the first consumer instance is selected from the consumer group.

3. The method as described in claim 2, characterized in that, Obtaining the second consumer identifier of the consumer group includes: Obtain the mapping relationship between the first consumer identifier and the topic set; Based on the mapping relationship, a set of topics corresponding to different first consumption identifiers is determined, and the set of topics includes at least one topic; From the topic sets corresponding to each of the first consumer identifiers, a target topic set including the second topic is determined, and the first consumer identifier corresponding to the target topic set is determined as the second consumer identifier.

4. The method as described in claim 3, characterized in that, The step of obtaining the mapping relationship between the first consumer identifier and the topic set includes: For each consumer instance, analyze the topics of messages that the consumer instance can consume; The topics of messages that each consumer instance can consume are clustered to obtain at least one cluster, and the topics in each cluster constitute a topic set; Generate cluster identifiers that correspond one-to-one with the clusters, use the cluster identifiers as the first consumption identifiers, and establish a mapping relationship between the first consumption identifiers and the topic set.

5. The method as described in claim 2, characterized in that, The number of second consumer identifiers is at least one, and the step of selecting the first consumer instance from the consumer group based on the first consumer identifier and the second consumer identifier includes: For each consumer instance in the consumer group, the corresponding first consumer identifier is matched with each second consumer identifier to obtain a matching result; When the matching result indicates that there is a second consumer identifier that matches the first consumer identifier, the corresponding consumer instance is determined as the first consumer instance.

6. The method as described in claim 1, characterized in that, The topic information is included in the consumption management information. Determining the first consumer instance in the consumer group that needs to be allocated messages includes: When the consumption management information includes a disable identifier, a second consumer instance that does not need to be allocated messages is selected from the consumer group based on the disable identifier; The disabled identifier is the instance identifier of a consumer instance that does not need to allocate messages; Based on the consumer instances in the consumer group other than the second consumer instance, the first consumer instance that needs to be allocated messages is determined.

7. The method as described in claim 6, characterized in that, The step of filtering out second consumer instances from the consumer group that do not need to be allocated messages based on the disabled identifier includes: Obtain the instance identifier of each consumer instance in the consumer group; The disabled identifier is matched with the instance identifier of each consumer instance to obtain the matching result; When the matching result indicates that the instance identifier matches the disabled identifier, the consumer instance corresponding to the instance identifier is taken as the second consumer instance; The method further includes: Unmount the partition that was historically allocated to the second consumer instance.

8. The method as described in claim 1, characterized in that, The topic information is included in the consumption management information. The step of allocating messages related to the second topic to the first consumer instance based on the consumption information of the first consumer instance includes: When the consumption management information includes partition information, the partition information is used to allocate the message of the second topic to the first consumer instance; The partition information is used to indicate the partition allocated to each of the first consumer instances.

9. The method as described in claim 1, characterized in that, After allocating messages of the second topic to the first consumer instance based on the consumption information of the first consumer instance, the method further includes: Obtain instance information for each consumer instance in the consumer group; The instance information includes at least one of the following: the instance identifier of the consumer instance, the group identifier of the consumer group, the topic subscribed to by the consumer instance, status information indicating whether the consumer instance is consuming normally, and partition information. A consumption report is generated based on the instance information of each consumer; wherein the consumption report is used to view the consumption status of each consumer instance.

10. The method as described in claim 1, characterized in that, The step of obtaining the topic information corresponding to the consumer group includes: In response to receiving consumption management information, the consumption management information is parsed to obtain topic information corresponding to the consumer group, wherein the consumption management information is configured by the administrator of the consumer group; Alternatively, the pre-set topic information can be retrieved from the storage structure.

11. The method as described in claim 10, characterized in that, The consumption management information also includes a disable identifier and partition information. The disable identifier is the instance identifier of a consumer instance that does not need to be allocated messages, and the partition information is used to indicate the partition allocated to each consumer instance. Determining the first consumer instance in the consumer group that needs to be allocated messages includes: Based on the disabled flag, a third consumer instance that cannot process messages is selected from the consumer group; The consumer instance other than the third consumer instance among the at least one consumer instance is designated as the fourth consumer instance; Obtain the first consumer identifier corresponding to each of the fourth consumer instances, and obtain the second consumer identifier of the consumer group; Based on the first consumer identifier and the second consumer identifier, the first consumer instance is selected from the fourth consumer instance; The step of allocating messages of the second topic to the first consumer instance based on the consumption information of the first consumer instance includes: Using the partition information, messages of the second topic are assigned to the first consumer instance.

12. The method as described in claim 1, characterized in that, After obtaining the topic information corresponding to the consumer group, the method further includes: When the topic information includes only the first topic, obtain the load information of each consumer instance in the consumer group; Based on the load information, a fifth consumer instance with a load less than or equal to the load threshold is selected from the consumer group; Based on the consumption information of the fifth consumer instance, the message of the first topic is assigned to the fifth consumer instance.

13. The method as described in claim 1, characterized in that, The number of consumer groups is multiple, and before obtaining the topic information corresponding to the consumer groups, the method further includes: The consumer groups are sorted to obtain a consumer group sequence; The step of obtaining the topic information corresponding to the consumer group includes: Based on the consumer group sequence, when the processing time point of the consumer group is reached, the topic information corresponding to the consumer group is obtained.

14. A message processing device, characterized in that, The device includes: The first acquisition module is used to acquire topic information corresponding to a consumer group, wherein the consumer group includes at least one consumer instance of the message; The determination module is used to determine the first consumer instance in the consumer group that needs to be allocated messages when the topic information includes a second topic with a higher priority than the first topic, wherein the first topic is the topic subscribed to by the consumer instance in the consumer group; The second acquisition module is used to acquire the consumption information of the first consumer instance, wherein the consumption information is used to indicate the topic of the message that the first consumer instance needs to process; The allocation module is used to allocate messages of the second topic to the first consumer instance based on the consumption information of the first consumer instance.

15. An electronic device, characterized in that, include: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the message processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, It stores computer-executable instructions or computer programs for inducing a processor to execute and implement the message processing method according to any one of claims 1 to 13.

17. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the message processing method according to any one of claims 1 to 13 is implemented.

Images