Message publishing method, message consumption method and related products
By introducing a proxy in the message queue system, using the mapping relationship between ordinary Topic and priority Topic, the problem of high complexity and cost of priority message management in the existing technology is solved, and efficient priority release and consumption of messages is achieved, reducing the The management complexity and cost of the business side.
Patent Information
- Application Number
- CN202410984554.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
The existing message queue system does not support priority messages, resulting in high-quality management and cost in business scenarios where high-quality messages are pushed and low-quality messages are prevented from starving.
By introducing agents on the production and consumer ends, using the mapping relationship between ordinary Topic and priority Topic, the priority release and consumption of messages is realized, reducing the complexity and cost of maintaining Topics on the business side. The specific method includes the production side agent determining the target topic based on the priority of the message and publishing it through the message queue server, and the consumer side agent determining the target topic based on the priority of the consumption request to consume.
The priority release and consumption of messages is realized, the complexity and cost of business-side maintenance topics is reduced, and the management process of priority messages is simplified.
Smart Images

Figure CN118945134A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of message queues, and in particular to a message publishing method, a message consuming method and related products. Background Art
[0002] Most message queue systems in the industry do not support priority messages, including mainstream message queue systems such as RocketMQ, Kafka, and Pulsar. However, in the face of a variety of business needs, priority messages are a very common application scenario in certain business scenarios. For example, red envelope messages and long link push messages need to ensure that red envelope messages are highly prioritized and can be preempted, while low-priority messages need to be prevented from being starved to death; for example, tasks that do not require priority can be queued up for execution, data queries can be queued up for execution, high-priority reports and frequently accessed report query requests can be queued up for execution, and so on.
[0003] However, currently, for businesses that need to design priority messages, topics (Topics) with the same number of priorities are generally created in advance based on the number of priorities. However, the business side needs to maintain multiple Topics, and the complexity of implementing priority messages is relatively high. Summary of the invention
[0004] The embodiments of the present application provide a message publishing method, a message consumption method and related products. For the business end, only ordinary topics need to be maintained to publish and consume messages according to priority, thereby reducing the complexity of priority messages and the cost of the business side.
[0005] In a first aspect, an embodiment of the present application provides a message publishing method, which is applied to a production-side agent, and the method includes:
[0006] Receive a first message publishing request from a production end, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic;
[0007] Based on the first priority and the general topic, determining a first target topic corresponding to the first priority;
[0008] A second message publishing request is sent to the message queue server to publish the message content to the first target topic, wherein the second message publishing request includes the message content and the first target topic.
[0009] In a second aspect, an embodiment of the present application provides a message consumption method, which is applied to a consumer-side agent, and the method includes:
[0010] Receiving a first message consumption request from a consumer, wherein the first message consumption request includes a common topic;
[0011] Obtaining a second priority corresponding to the first message consumption request;
[0012] Based on the second priority and the general topic, determining a second target topic corresponding to the second priority;
[0013] Consume messages based on the second target topic.
[0014] In a third aspect, an embodiment of the present application provides a production-end agent, including: a transceiver unit and a processing unit;
[0015] The transceiver unit is used to receive a first message publishing request from a production end, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic;
[0016] The processing unit is configured to determine a first target topic corresponding to the first priority based on the first priority and the common topic;
[0017] The transceiver unit is further used to send a second message publishing request to the message queue server to publish the message content to the first target topic, wherein the second message publishing request includes the message content and the first target topic.
[0018] In a fourth aspect, an embodiment of the present application provides a consumer-end agent, including: a transceiver unit and a processing unit;
[0019] The transceiver unit is used to receive a first message consumption request from a consumer end, wherein the first message consumption request includes a common topic;
[0020] The processing unit is used to obtain a second priority corresponding to the first message consumption request;
[0021] Based on the second priority and the general topic, determining a second target topic corresponding to the second priority;
[0022] Consume messages based on the second target topic.
[0023] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method described in the first aspect or the second aspect.
[0024] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the method described in the first aspect or the second aspect.
[0025] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer is operable to cause the computer to execute the method described in the first aspect or the second aspect.
[0026] Implementing the embodiments of the present application has the following beneficial effects:
[0027] It can be seen that in the embodiment of the present application, when a priority message needs to be implemented for a certain business, when the production end sends a message publishing request, it only needs to carry the ordinary Topic and the priority of the message. The production end agent will determine the priority topic corresponding to the priority based on the ordinary Topic and the priority, and then publish the message to the priority topic through the message queue server, thereby realizing the release of the message according to the priority. Therefore, the business side only needs to maintain an ordinary Topic to realize the release of priority messages, which reduces the cost of maintaining Topics on the business side and the complexity of publishing priority messages. When the consumer end needs to consume messages from the business, it only needs to carry an ordinary Topic. The consumer end agent will determine the priority of the consumer end's consumption this time, and then consume messages for the consumer end from the priority topic corresponding to the priority, thereby realizing the consumption of messages according to the priority of the message. Therefore, the business side only needs to maintain an ordinary Topic to realize the consumption of priority messages, which reduces the cost of maintaining Topics on the business side and the complexity of consuming priority messages. Therefore, when implementing priority messages, the complexity of the business side is relatively low, and the cost of the business side is also relatively low. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0029] Figure 1 A schematic diagram of a message queue system provided in an embodiment of the present application;
[0030] Figure 2 A flowchart of a message publishing method provided in an embodiment of the present application;
[0031] Figure 3 A schematic diagram of a Topic management interface provided in an embodiment of the present application;
[0032] Figure 4A flow chart of a message consumption method provided in an embodiment of the present application;
[0033] Figure 5 A flowchart of a message processing method provided in an embodiment of the present application;
[0034] Figure 6 A schematic diagram of message processing provided in an embodiment of the present application;
[0035] Figure 7 A flowchart of another message publishing method provided in an embodiment of the present application;
[0036] Figure 8 A schematic diagram of a message release provided in an embodiment of the present application;
[0037] Fig. 9 A schematic diagram of a production-end proxy provided in an embodiment of the present application;
[0038] Fig.10 A schematic diagram of a consumer-side proxy provided in an embodiment of the present application;
[0039] Fig.11 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0041] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0042] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0043] In order to facilitate understanding of the technical solution of this application, the relevant technologies involved in this application are first introduced.
[0044] Topic type: There are two types of topics involved in this application, one is a common topic and the other is a priority topic.
[0045] Among them, the priority of messages published to priority topics is higher than that of messages published to ordinary topics. When consuming messages, messages from priority topics are consumed first. At least most message consumption needs to consume messages from priority topics first.
[0046] Specifically, for some business scenarios, the messages in the business scenarios need to be prioritized. For such business, the above two types of Topics need to be deployed so that the messages can be processed according to the priority of the messages. For example, for the query business, some query requests are high priority (for example, the boss's query request), while some query requests are low priority (for example, the query request of ordinary employees). In other words, different query requests in the query business need to be prioritized. Therefore, it is necessary to deploy two types of Topics for the query business, publish high-priority query requests to priority Topics, and publish low-priority query requests to ordinary Topics. In this way, when the consumer consumes, it will first consume query requests from the priority Topic, so that the consumer can prioritize high-priority query requests and return query results for high-priority query requests. For some business scenarios, the messages in the business scenarios do not need to be prioritized. For such business, there is no need to deploy priority Topics, only ordinary Topics need to be deployed. For example, for the user behavior analysis business, for the user's behavior log, since the user's behavior log is generally not used immediately, there is no priority distinction between the behavior logs of different users, and there is no need to limit which user's behavior logs need to be consumed first. Therefore, for this business scenario, only a common Topic needs to be deployed. After the production end produces the user's behavior log, it will publish the behavior log to the common Topic according to the message publishing logic. The consumer end consumes different users' behavior logs from the common Topic according to the consumption logic. This application mainly takes the business of deploying two types of Topics as an example to illustrate.
[0047] Specifically, when some messages in the business have priorities, the priority of the message can be subdivided into N priorities, priority 1, priority 2, priority 3, ..., priority N, and the level indication method for the priority can be indicated by numbers, for example, the numbers 1, 2, 3, ..., N respectively indicate the levels of priority 1, priority 2, priority 3, ..., priority N, and the smaller the number, the higher the level of priority. In addition, for each priority, a corresponding priority Topic can be deployed for each priority to support the publication and consumption of messages of this priority. In addition, there will be some messages in this business that have no priority distinction. For such messages, this application will also deploy a common Topic to support the publication and consumption of these messages. It can be understood that a message without priority can also be understood as the message with the lowest priority, so the above-mentioned common Topic can also be understood as a priority Topic, but the priority corresponding to the priority Topic is the lowest or has no priority. In this application, the priority of a common Topic is indicated by the number 0.
[0048] Therefore, this application can essentially be understood as the deployment of (N+1) priority topics for the business. In order to uniformly describe any topic mentioned later, both ordinary topics and priority topics can be referred to as priority topics.
[0049] For the convenience of description, Figure 1 As shown, this application uses Topic 0, Topic 1, ..., Topic N to represent the above-mentioned N+1 priority Topics respectively. Specifically, Topic 0 represents a normal Topic, Topic 1, ..., Topic N represent N priority Topics corresponding to priority 1, priority 2, ..., priority N respectively.
[0050] In addition, each Topic in the present application includes multiple queues, and in order to better limit the flow and better execute the balance strategy when subscribing, the number of queues of priority Topics and common Topics is the same. Moreover, the multiple queues of each Topic are evenly distributed on each message queue server of the message queue system.
[0051] In addition, the application distributes the queue on the message queue server, which can also be referred to as the message queue server managing the queue.
[0052] In addition, the message queue system of the present application can be any type of message queue system, such as Apache Kafka, RabbitMQ, Apache ActiveMQ, RocketMQ, Amazon Simple Queue Service. This application mainly uses RocketMQ as an example to illustrate the message queue system, but does not limit the type of message queue system.
[0053] See also Figure 1 , Figure 1 A schematic diagram of a message queue system provided in an embodiment of the present application.
[0054] like Figure 1 As shown, the message queue system includes a producer, a consumer, a producer proxy (producer proxy, pproxy) cluster, a consumer proxy (consumer proxy, cproxy) cluster, a message queue server (Broker) cluster, and a console.
[0055] pproxy cluster includes multiple production-side proxy pproxy, such as Figure 1 As shown in the figure, the pproxy cluster includes pproxy 1, pproxy 2, ..., pproxy n; the cproxy cluster includes multiple consumer-side proxies, such as Figure 1 As shown, the cproxy cluster includes cproxy 1, cproxy 2, ..., cproxy m; the Broker cluster includes multiple Brokers, such as Figure 1 Broker 1, ..., Broker b are shown, and this application is mainly described by taking one Broker as an example.
[0056] The console provides a Topic management interface to facilitate the creation of Topic 0, Topic 1, ..., Topic N.
[0057] Combine the following Figure 1 The message queue system shown introduces the message publishing process of this application.
[0058] For example, Figure 1 As shown, when the producer produces a message, it determines the priority of the message and then encapsulates the message into a first message publishing request, wherein the first message publishing request includes the message content, the first priority of the message content, and the common topic. Then, the producer selects a pproxy from the pproxy cluster and sends the first message publishing request to the pproxy, as shown in FIG. Figure 1 As shown, the production end selects pproxy1 and sends a first message publishing request to pproxy1. Accordingly, pproxy parses the first message publishing request, obtains the common topic and the priority, and determines the first target topic corresponding to the first priority based on the first priority and the common topic. Figure 1 As shown, when the priority is priority 1, the first target topic can be determined to be Topic 1.
[0059] The pproxy then generates a second message publishing request based on the first target topic and selects a Broker from the Broker cluster, i.e., a message queue server (such as Figure 1 Broker 1) shown in the figure then sends a second message publishing request to the message queue server, wherein the second message publishing request includes the message content and the first target topic. Accordingly, after receiving the second message publishing request, the message queue server will parse the second message publishing request, obtain the first target topic, and publish the message content to the first target topic, that is, publish the message content to the queue of the first target topic. Figure 1As shown in the figure, Broker 1 publishes the message content to the first queue of Topic 1.
[0060] It can be seen that in the embodiment of the present application, when a priority message is required for a certain business, when the production end sends a message publishing request, it only needs to carry the ordinary Topic and the priority of the message. The production end agent will determine the priority topic corresponding to the priority based on the ordinary Topic and the priority, and then publish the message to the priority topic through the message queue server, thereby realizing the publishing of messages according to priority. Therefore, the business side only needs to maintain an ordinary Topic to realize the publishing of priority messages, which reduces the cost of maintaining the Topic on the business side and the complexity of publishing priority messages.
[0061] Combine the following Figure 1 The message queue system shown introduces the message consumption process of this application.
[0062] like Figure 1 As shown in the figure, when the message queue server publishes a message in any Topic it maintains, it pushes the message to the cproxy that subscribes to the Topic. The cproxy caches the message and waits for the consumer to consume it. It can be understood that each cproxy caches the message in any Topic. When the consumer consumes the message, it selects a cproxy from the cproxy cluster to provide proxy services, such as Figure 1 As shown, cproxy 1 is selected to provide proxy service. Then, the consumer sends a first message consumption request to the cproxy, wherein the first message consumption request includes a common topic. Then, the cproxy obtains the second priority corresponding to the first message consumption request; based on the second priority and the common topic, determines the second target topic corresponding to the second priority, and finally the cproxy consumes the message based on the second target topic, that is, pulls the message for the consumer from the second target topic.
[0063] It can be seen that in the embodiments of the present application, when priority messages need to be implemented for a certain business, when the consumer needs to consume messages from the business, it only needs to carry an ordinary Topic. The consumer agent will determine the priority of the consumer's consumption this time, and then consume messages for the consumer from the priority topic corresponding to the priority, thereby realizing the consumption of messages according to the priority of the messages. Therefore, the business side only needs to maintain an ordinary Topic to realize the consumption of priority messages, reducing the cost of maintaining the Topic on the business side and the complexity of consuming priority messages.
[0064] See also Figure 2 , Figure 2A flow chart of a message publishing method provided in an embodiment of the present application. Figure 1 The message queue system shown. The method includes but is not limited to the following steps:
[0065] 201: The producer sends a first message publishing request to the producer proxy, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic.
[0066] It should be noted that each message queue server in the above-mentioned message queue system is distributed with a common topic and a plurality of priority topics corresponding to the common topic, wherein the plurality of priority topics are created based on the common topic and the number of priorities.
[0067] Exemplarily, before executing the message publishing of the present application, each message queue server in the message queue system needs to first create the common topic and multiple priority topics corresponding to the common topic.
[0068] Exemplarily, the console will provide a Topic management interface, through which the user can create the above-mentioned common Topic and multiple priority Topics. Specifically, the user can create the above-mentioned common Topic and multiple priority Topics by inputting the configuration parameters of the common Topic through the Topic management interface, where, for example, Figure 3 As shown, the configuration parameters include the type of message queue, the name of a common Topic, the number of partitions, the number of replicas, the number of priorities, the current limit value, etc. This application mainly takes the configuration parameters of the name of a common Topic and the number of priorities as an example for explanation.
[0069] Correspondingly, after the console obtains the name of the common topic entered by the user and the number of priorities N, it can simultaneously create the common topic and N priority topics corresponding to the common topic.
[0070] Exemplarily, the console sends a common topic creation request corresponding to the common topic to each message queue server based on the name of the common topic, wherein the common topic creation request includes the name of the common topic. Each message queue server creates the common topic in response to the common topic creation request. Then, the console generates priority topics corresponding to each priority level based on the name of the common topic, the number of priorities, and the pre-configured priority topic generation rules. Then, for each priority level, a priority topic creation request corresponding to the priority level is sent to each message queue server, wherein the priority topic creation request includes the name of the priority topic of each priority level. Accordingly, each message queue server creates priority topics corresponding to each priority level in response to the priority topic creation request corresponding to each priority topic, and obtains N priority topics.
[0071] It can be seen that in this application, while creating a common Topic, multiple priority Topics corresponding to the common Topic are simultaneously created, and the creation of all Topics is completed at one time, thereby improving the efficiency of Topic creation.
[0072] It can be understood that if the priority of the business needs to be expanded or reduced later, it is only necessary to adjust the number of priorities in the Topic management interface to complete the expansion or reduction of the priority of the business.
[0073] It should be noted that before sending the first message publishing request, the producer first selects the above-mentioned producer proxy from the pproxy cluster. For the sake of distinction, the present application may refer to the selected producer proxy as the target producer proxy.
[0074] Optionally, the producer can randomly select a pproxy from the pproxy cluster as the target producer proxy. Optionally, the producer can obtain the status of each pproxy in the pproxy cluster, and select the target producer proxy from the pproxy cluster based on the status of each pproxy. For example, the producer can select the pproxy with the best network status from the pproxy cluster as the target producer proxy.
[0075] It should be understood that the above method of selecting the target production-side agent is only for illustration, and the present application does not limit the method of selecting the target production-side agent. The production end can select the target production-side agent in any feasible manner.
[0076] Since the present application aims to publish messages according to their priorities, the production end also needs to indicate the priority of the message, that is, to indicate the priority of the message content in the first message publishing request. Exemplarily, the production end can indicate the priority of the message through one or more fields. The present application mainly uses one field as an example to illustrate the priority of the message content. Exemplarily, a priority indication field is pre-configured in the message publishing request, such as the Priority field, and the priority of the message content is indicated by the value of the field. For example, if the value of the Priority field is 1, it indicates that the priority of the message content is priority 0. It can be understood that when the value of the Priority field is 0, it indicates that the message content is a normal message.
[0077] It should be noted that the priority of the message produced by the production end can be determined based on the source of the message. For example, if the message is a query request, the priority corresponding to the query request can be determined based on the identity information of the user who initiated the query request.
[0078] Therefore, for the message content of this application, the production end can determine that the priority corresponding to the message content is the first priority based on the source of the message content, and can encapsulate the message content, the common topic and the first target priority to obtain the first message publishing request. For example, the first priority is priority 1, and the encapsulated first message publishing request is: Msg request1 (Topic = Topic A, content = content1, Priority = 1), where Msg1 is the first message publishing request, Topic A is the first target topic, content1 is the message content, and Priority = 1 is used to indicate that the priority of the message content is the first priority. Finally, the production end sends the first message publishing request to the production end agent.
[0079] 202: The production-end agent determines a first target topic corresponding to the first priority based on the first priority and the common topic.
[0080] Exemplarily, after receiving the first message publishing request, the production-end proxy parses the first message publishing request to obtain the common topic and the first priority.
[0081] Furthermore, if the first priority is less than the minimum priority (i.e., less than 0) or greater than the maximum priority (i.e., greater than Priority max), where Priority max is the maximum priority, the minimum priority is used as the first priority, that is, the message content is published to the normal Topic. Of course, if the first priority is greater than the maximum priority, the maximum priority can also be used as the first priority. In the case of an exception to the first priority, this application does not limit the method of processing the first priority, as long as the first priority is between the minimum priority and the maximum priority after the exception is processed.
[0082] Further, if the first priority is the priority corresponding to the ordinary topic, that is, the first priority is the lowest priority, the ordinary topic will be used as the first target topic; if the first priority is not the priority corresponding to the ordinary topic, the first target topic will be determined based on the pre-configured priority topic generation rules, as well as the first priority and the ordinary topic.
[0083] Exemplarily, the first target theme is determined based on the pre-configured priority theme generation rules, the common theme and the first priority. Specifically, the first preset characters, the common theme and the first priority are spliced together to obtain the first target theme. In addition, in order to be able to identify the spliced content in the first target theme, special characters can be added between the first preset characters, the common theme and the first priority to splice the above-mentioned first target theme.
[0084] For example, if the common topic is Topic A, the first priority is 1, and the first preset character is LEVEL, for example, the special character is %, or #, where the special character is a character that is not used alone when naming a topic. Therefore, the first preset character, the common topic, and the first priority are concatenated to obtain the first target topic, namely %LEVEL%TopicA#1.
[0085] Furthermore, after determining the first target topic, if the first target topic is a common topic, the first message publishing request is used as the second message publishing request, and the second message publishing request is sent to the message queue server, that is, the first message publishing request is forwarded to the message queue server; if the first target topic is not a common topic, a second message publishing request is generated, wherein the second message publishing request includes the message content and the first target topic, and then the second message publishing request is sent to the message queue server.
[0086] Furthermore, the second message publishing request can carry the priority of the message, and the second message publishing request can be generated by replacing the common topic in the first message publishing request with the first target topic. There is no need to modify the format of the message publishing request, and the changes are relatively small.
[0087] It should be noted that before sending the second message publishing request to the message queue server, the production agent needs to select a message queue server from the message queue server cluster. For easy distinction, the message queue server selected by the production agent can be called the target message queue server.
[0088] Optionally, the production-side proxy can randomly select a message queue server from the message queue server cluster as the target message queue server. Optionally, the production-side proxy obtains the load of each Broker and selects the target message queue server based on the load of each Broker. For example, the Broker with the least load is used as the target Broker. This application does not limit the method of obtaining the target message queue server.
[0089] 203: The production-end agent sends a second message publishing request to the message queue server, wherein the second message publishing request includes the message content and the first target topic.
[0090] 204: The message queue server publishes the message content to the first target topic.
[0091] Exemplarily, after receiving the second message publishing request, the message queue server parses the second message publishing request to obtain the message content and the first target topic, and then publishes the message content to multiple queues of the first target topic managed by the message queue. For example, the message queue server selects a queue from multiple queues of the first target topic managed by the message queue server through polling, or randomly selects a queue from the multiple queues and publishes the message content to the queue.
[0092] It can be seen that in the embodiment of the present application, when a priority message is required for a certain business, when the production end sends a message publishing request, it only needs to carry the ordinary Topic and the priority of the message. The production end agent will determine the priority topic corresponding to the priority based on the ordinary Topic and the priority, and then publish the message to the priority topic through the message queue server, thereby realizing the publishing of messages according to priority. Therefore, the business side only needs to maintain an ordinary Topic to realize the publishing of priority messages, which reduces the cost of maintaining the Topic on the business side and the complexity of publishing priority messages.
[0093] See also Figure 4 , Figure 4 A flow chart of a message consumption method provided in an embodiment of the present application. The method is applied to the above-mentioned message queue system. The method includes but is not limited to the following steps:
[0094] 401: The consumer sends a first message consumption request to the consumer proxy, wherein the first message consumption request includes a common topic.
[0095] Exemplarily, when a consumer needs to consume a message, it will select a consumer proxy from the consumer proxy cluster to consume the message. Similarly, the cproxy is randomly selected by the consumer from the cproxy cluster, or the consumer is selected based on the status of each cproxy in the cproxy cluster, for example, the cproxy with the best network condition is selected as the cproxy. For the sake of distinction, the present application can use the cproxy selected by the consumer as the target consumer proxy.
[0096] Among them, the first message consumption request also includes a first quantity, wherein the first quantity is used to indicate that at most the first quantity of messages are consumed. If the first quantity of messages can be consumed, the first quantity of messages are consumed. If the number of consumable messages is less than the first quantity, as many messages as can be consumed are consumed.
[0097] 402: The consumer-end agent obtains a second priority corresponding to the first message consumption request.
[0098] Exemplarily, the consumer-side agent determines the second priority based on the consumption strategy.
[0099] It should be noted that for different consumption strategies, the consumer-side agent determines the second priority in different ways.
[0100] Optionally, the consumption strategy includes an absolute priority strategy, that is, each time the message is consumed from the topic with the highest priority first, ensuring that the message with the highest priority is always consumed first. Therefore, the highest priority is used as the second priority corresponding to the first message consumption request. For example, if the highest priority is Priority max, Priority max is used as the second priority.
[0101] Optionally, the consumption strategy includes an anti-starvation strategy, that is, to ensure that messages from high-priority topics are consumed first as much as possible, but also to ensure that messages in low-priority topics are not starved and can be consumed.
[0102] Exemplarily, a preset weight is configured for each priority level, and based on the preset weight corresponding to each priority level, a random number is generated for the first message consumption request, and the priority level corresponding to the random number is used as the second priority level, wherein the preset weight of the high priority level is greater than the preset weight of the low priority level. In this way, the probability of generating a random number of a high priority level is greater than that of a low priority level. Therefore, messages will be consumed from high-priority topics as much as possible, that is, messages in high-priority topics will be consumed first. However, there is a certain probability of generating random numbers corresponding to low priority levels, and messages will also be consumed from low-priority topics, thereby ensuring that messages in low-priority topics will not be starved to death.
[0103] It can be seen that by configuring preset weights for priorities, high-priority messages can be consumed first without designing complex current limiting strategies. Moreover, a random number is used to determine the queue from which the message is consumed, so that the consumed messages are consumed from topics of the same priority as much as possible, reducing the difficulty of message consumption and facilitating engineering implementation.
[0104] For example, there are three levels of priority: high, medium and low, and the preset weights configured for the three levels are 70%, 25% and 5% respectively. In this way, when the consumer consumes 100 messages, the random numbers generated for 70 of the consumptions correspond to high priority, the random numbers generated for 25 of the consumptions correspond to medium priority, and the random numbers generated for 5 of the consumptions correspond to low priority. In this way, 70 consumptions will be made from the logical queue group corresponding to high priority, 25 consumptions will be made from the topic corresponding to medium priority, and 5 consumptions will be made from the topic corresponding to low priority. This ensures that high-priority messages are consumed first, and low-priority messages can also be consumed without being starved to death.
[0105] Optionally, the message consumption strategy also includes a total consumption strategy. The consumer-end agent then uses each priority as the second priority. Specifically, a preset weight is configured for each priority, wherein the preset weight of the high priority is greater than the preset weight of the low priority. Then, based on the preset weight corresponding to each priority and the first quantity, the number of messages consumed from each priority topic is determined. For example, the first quantity is 100, and the preset weights of high, medium and low priorities are 60%, 30% and 10% respectively. Based on the preset weights, it is determined that 60 messages need to be consumed from the high-priority topic, 30 messages need to be consumed from the medium-priority topic, and 10 messages need to be consumed from the low-priority topic.
[0106] 403: The consumer-end agent determines a second target topic corresponding to the second priority based on the second priority and the common topic.
[0107] Exemplarily, if the second priority is the priority corresponding to the common topic, the common topic is used as the second target topic, that is, messages are consumed from the common topic first; if the second priority is not the priority corresponding to the common topic, the second target topic is determined based on the pre-configured priority topic generation rule, the second priority and the common topic. The consumer-side proxy generates the target topic in a similar manner to the above-mentioned producer-side proxy, which will not be described again.
[0108] 404: The consumer-end proxy consumes messages based on the second target topic.
[0109] First of all, the consumer-side proxy will create a corresponding buffer for each topic of the business (including priority topics and ordinary topics) to be used to cache the messages pulled from each topic. The consumer-side proxy will first subscribe to each topic of the business created by the message queue server. In this way, when the message queue server publishes the message content to a topic (that is, publishes to the queue of a topic), the consumer-side proxy can sense that there is a new message published in the topic, and will pull the message from the topic, and cache the pulled message to the buffer created for the topic, waiting for subsequent consumer consumption.
[0110] It should be noted that the consumer-side agent can subscribe to the Topic when the consumer-side agent is started, or it can subscribe to the Topic based on actual needs. This application mainly uses the example of the consumer-side agent subscribing to the Topic when it is started to illustrate. Secondly, when multiple consumer-side agents subscribe to the topic of the business created by the message queue server, multiple consumer-side agents subscribe to part or all of the queues in each topic through the rebalance strategy, and the specific process will not be described. In general, the consumer-side agent can subscribe to each topic, so that it can pull messages from each topic for caching.
[0111] Furthermore, after determining the second target topic, the consumer-end agent can consume messages from the cache corresponding to the second target topic. Specifically, if the number of messages cached in the cache area corresponding to the second target topic is greater than or equal to the first number, the first number of messages are pulled from the cache area corresponding to the second target topic for the consumer end and returned to the consumer end; if the number of messages cached in the cache area corresponding to the second target topic is less than the first number, all messages cached in the cache area corresponding to the second target topic are pulled out, and then continue to pull messages from the cache area corresponding to the third target topic for the consumer end until the number of pulled messages reaches the first number. Finally, the first number of messages pulled out are returned to the consumer end.
[0112] For example, if the second priority is priority 1, messages will be pulled from the priority topic corresponding to priority 1, that is, the cache area corresponding to Topic 1. If the first number is 32, and 15 messages are cached in the cache area corresponding to Topic 1, 10 messages are cached in the cache area corresponding to Topic 2, and 9 messages are cached in the cache area corresponding to Topic 3, then 15 messages will be pulled from the cache area corresponding to Topic 1, 10 messages will be pulled from the cache area corresponding to Topic 2, and 7 messages will be pulled from the cache area corresponding to Topic 3, thereby pulling 32 messages and returning them to the consumer.
[0113] It should be noted that if the consumer-side proxy pulls messages from the topics corresponding to the multiple priorities in order from high to low priority, starting from the second target topic corresponding to the second priority; if the total number of messages pulled from the cache corresponding to the topics of each priority is still less than the first number, all the pulled messages will be returned to the consumer. For example, if the first number is 32 and the second priority is priority 1, then starting from the cache corresponding to the topic with the highest priority, until the cache corresponding to the topic with the lowest priority, the total number of messages pulled is 26, and the 26 messages are returned to the consumer.
[0114] It can be seen that in this application, empty pull optimization is designed for message consumption, that is, when the number of messages pulled from the current priority topic is insufficient or even 0, messages can be pulled from low-priority topics to avoid not returning any messages to the consumer end.
[0115] In one embodiment of the present application, for the cache area corresponding to each topic, the cache areas corresponding to multiple topics can be combined into one logical cache area, that is, the whole is used as a logical cache area. In this way, when the consumer-side proxy builds a cache area for the topic of a service with priority messages, it can reuse the logic code of the service without priority messages, without configuring a piece of code for the cache area corresponding to each topic, thereby improving the reuse of the code.
[0116] It can be seen that in the embodiments of the present application, when priority messages need to be implemented for a certain business, when the consumer needs to consume messages from the business, it only needs to carry an ordinary Topic. The consumer agent will determine the priority of the consumer's consumption this time, and then consume messages for the consumer from the priority topic corresponding to the priority, thereby realizing the consumption of messages according to the priority of the messages. Therefore, the business side only needs to maintain an ordinary Topic to realize the consumption of priority messages, reducing the cost of maintaining the Topic on the business side and the complexity of consuming priority messages.
[0117] The following takes the publication of the above message content and the consumption of the above message content, that is, the second priority is the first priority, and the second target topic is the first target topic as an example to introduce the message processing process of this application as a whole.
[0118] See also Figure 5 , Figure 5 A flowchart of a message processing method provided in an embodiment of the present application. The method is applied to the above-mentioned message queue system. The method includes but is not limited to the following steps:
[0119] 501: The producer sends a first message publishing request to the producer proxy, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic.
[0120] 502: The production-end agent determines a first target topic corresponding to the first priority based on the first priority and the common topic.
[0121] 503: The production-end agent sends a second message publishing request to the message queue server, wherein the second message publishing request includes the message content and the first target topic.
[0122] 504: The message queue server publishes the message content to the first target topic.
[0123] 505: The consumer-side agent consumes the message content from the first target topic of the message queue server.
[0124] 506: The consumer-end proxy caches the message content into a cache area corresponding to the first target topic.
[0125] 507: The consumer sends a first message consumption request to the consumer proxy, wherein the first message consumption request includes a common topic.
[0126] 508: The consumer-end proxy obtains a second priority corresponding to the first message consumption request, where the second priority is the first priority.
[0127] 509: The consumer-end agent determines a second target topic corresponding to the second priority based on the second priority and the common topic, wherein the second target topic is the first target topic.
[0128] 510: The consumer-end proxy pulls the message content from the cache area corresponding to the first target topic.
[0129] 511: The consumer-end proxy sends the message content to the consumer-end.
[0130] It should be noted that in this embodiment, the process of publishing messages in steps 501-506 is similar to the process of publishing messages described above, and will not be described again; the process of consuming messages in steps 507-511 is similar to the process of consuming messages described above, and will not be described again.
[0131] Combine the following Figure 6 The message processing process of this application is illustrated by taking the common Topic as Topic A, N priority Topics as %LEVEL%TopicA#1, %LEVEL%TopicA#2, ..., %LEVEL%TopicA#N, and the consumer end consuming 32 pieces of data as an example.
[0132] First, it should be noted that Figure 6 As shown, cproxy creates Buffer 0, Buffer 1, ..., Buffer N for Topic A, %LEVEL% Topic A#1, %LEVEL% Topic A#2, ..., %LEVEL% Topic A#N, respectively, and subscribes to Topic A, %LEVEL% Topic A#1, %LEVEL% Topic A#2, ..., %LEVEL% Topic A#N, respectively.
[0133] like Figure 6 As shown, the production end sends a first message publishing request to pproxy, namely, Msg request 1 (Topic = Topic A, content = content1, Priority = 1). Then, pproxy determines the first target topic corresponding to the first priority, namely, %LEVEL% Topic A#1, based on the first priority (ie, Priority = 1) and the common topic Topic A. Then, the common topic in the first message publishing request is replaced with the first target topic to obtain the second message publishing request, namely, Msg request 2 (Topic = %LEVEL% Topic A#1, content = content1, Priority = 1). Then, pproxy sends the second message publishing request to the message queue server Broker.
[0134] Then, Broker takes the message content from the second message publishing request Msg request 2 and publishes the message content to the first target topic %LEVEL% Topic A#1, as shown in Figure 6 As shown, the message content is published to the first queue of %LEVEL% Topic A#1.
[0135] Furthermore, since cproxy subscribes to %LEVEL% Topic A#1, when Broker publishes the message content to the queue, cproxy pulls the message content from the queue and caches the message content in the buffer area corresponding to %LEVEL% Topic A#1, namely, Buffer 1.
[0136] Furthermore, when the consumer needs to consume 32 messages, it can send a first consumption request to cproxy, wherein the first consumption request includes 32, indicating to pull 32 messages from cproxy. Then cproxy determines to consume messages from %LEVEL% Topic A#1 first, that is, to pull messages from Buffer 1 first; Figure 6 As shown in the figure, 10 messages are pulled from Buffer 1, that is, 10 messages are returned (Res), and there are 22 messages left. Then, messages are pulled from Buffer 2. It can be seen that 6 messages are pulled from Buffer 2, and then 8 messages are pulled from Buffer N and 2 messages are pulled from Buffer 0 in sequence. That is, all the messages cached in the buffer areas corresponding to all topics are pulled out, and a total of 26 messages are pulled. At this time, 26 messages will be returned (Res) to the consumer.
[0137] In one embodiment of the present application, the above-mentioned business also needs to implement the function of delayed messages, that is, to delay the release and consumption of some messages. For example, for e-commerce business scenarios, there will be promotional activities at a specific time point in the future, so it is necessary to set up messages related to the promotional activities in advance (for example, coupon messages), and publish the messages at that specific time point; at the same time, when the specific time point arrives, other messages will be produced in the business, and the priorities of these messages need to be distinguished. For example, when a specific time point arrives, the priority of coupon messages and order messages is higher than that of logistics messages. Therefore, at a specific time point, it is necessary to consume messages according to the priority of the messages to ensure the normal execution of the business.
[0138] Therefore, in some business scenarios, some messages need to be processed in a delayed manner and in a prioritized manner. The following will describe in detail the publishing process for messages that require both priority consumption and delayed consumption in conjunction with the accompanying drawings.
[0139] See also Figure 7 , Figure 7 A flow chart of another message publishing method provided in an embodiment of the present application. The method is applied to the above-mentioned message queue system. The contents repeated in the embodiment of the method and the above-mentioned message publishing embodiment are not described again. The method includes but is not limited to the following steps:
[0140] 701: The producer sends a first message publishing request to the producer proxy, wherein the first message publishing request includes message content, a first priority of the message content, a delay time, and a common topic.
[0141] It should be noted that when creating a normal Topic, a priority Topic and multiple delayed Topics will be created simultaneously. The synchronous creation of the priority Topic can be similar to the above creation method, such as Figure 8 As shown in the figure, the priority topics created are Topic 0, Topic 1, ..., Topic N, which will not be described again.
[0142] For delayed topics, you can enter the number M of delayed topics in the Topic management interface, that is, create M delayed topics. Specifically, the console generates M delayed topics based on the common topics, the number of delayed topics, and the delay topic generation rules. Similarly, the console can splice the second preset character, the common topic, and the serial number of each delayed topic to obtain M delayed topics. For example, the common topic is Topic A, the second preset character is delay, and the serial numbers of the delay topics are 1, 2, 3, ..., M, then the M delayed topics obtained after splicing are %delay%Topic A#1, %delay%Topic A#2, ..., %delay%Topic A#M. For the convenience of description. As Figure 8 As shown in the present application, the M delay topics may be referred to as delay Topic 1, delay Topic 2, ..., delay Topic M. Then for each delay topic, the console sends a delay Topic creation request to each message queue server respectively, wherein the delay Topic creation request includes the delay topic, thereby completing the creation of each delay topic.
[0143] In one embodiment of the present application, if the delay time of the message needs to be indicated, it is necessary to carry a delay time indication field in the message publishing request. For example, the delay time can be carried in the message publishing request, and the delay time of the message is indicated by the value of the delay time field. Therefore, in the case of carrying the delay time, the above-mentioned first message publishing request is still used, and the above-mentioned first message publishing request is Msg request 1 (Topic = Topic A, content = content1, Priority = 1, delay time = t1). Among them, t1 is the delay time.
[0144] 702: The production-end agent determines a first target delayed topic corresponding to the message content based on the number of delayed topics and the common topic.
[0145] Exemplarily, the production end generates a first serial number based on the number of delay topics. For example, the first serial number is randomly generated based on the number of the delay topics. Then, the first serial number is used as the serial number of the first target delay topic, and the first target delay topic is determined based on the common topic, the first serial number and the delay topic generation rule. For example, if the first serial number is 1, then the first target delay topic is %delay%Topic A#1. The way in which the production end agent generates the first target delay topic is similar to the way in which the console generates the delay topic, which will not be described again.
[0146] 703: The production-end agent sends a third message publishing request to the message queue server, wherein the third message publishing request includes the message content, the first priority, the delay time, and the first target delay topic.
[0147] Exemplarily, the production-side proxy replaces the first target delayed topic with the common topic in the first message publishing request to generate a third message publishing request. Figure 8 As shown, after selecting the message queue server Broker, the production-end agent sends the third message publishing request to the message queue server.
[0148] 704: The message queue server publishes the message content to the first target delay topic in the form of a key-value pair, wherein the key in the key-value pair includes the expiration time of the message content, the first priority, and the first target delay topic, the expiration time is determined based on the delay time, and the value in the key-value pair is the message content.
[0149] Exemplarily, after receiving the third message publishing request, the message queue server can parse the message content, the delay time of the message content, the first priority and the first target delay topic, and then publish the message content to the first target delay topic, that is, cache the message content to the database corresponding to the first target delay topic.
[0150] Furthermore, when caching the message content, the message queue server will cache the message content in the form of a key-value pair, so the message queue server will first construct the key of the message content based on the delay time of the message content, the first priority, and the first target delay theme. Exemplarily, the expiration time of the message content is determined based on the delay time and the current time, and then the key is constructed based on the expiration time, the first priority, and the first target delay theme. Specifically, the expiration time, the first priority, and the first target delay theme can be concatenated to obtain the key.
[0151] To facilitate machine recognition, special characters can be added to each field in the key to isolate the fields in the key. The special characters added at the end of each field are characters that are not used in any field. For example, "#" can be used as a special character for isolation. The key is #timestamp#Priority#delay Topic, where timestamp is the expiration time, Priority is the priority of the message content, and delay Topic is the first target delay topic.
[0152] Furthermore, in order to better distinguish each message, the unique identifier UUID of each message can be spliced into the key, and the key can be #timestamp#Priority#delay Topic#UUID. Of course, other parameters can also be spliced into the key, such as the shard number storing the message content, etc.
[0153] It should be noted that no matter how many fields are concatenated in the key, the expiration time is concatenated at the first position of the key in this application.
[0154] Therefore, after constructing the above key, the message content can be used as the value, the key-value pair, ie, (#timestamp#Priority#delay Topic#UUID, content1) can be cached in the database, and the message content can be published to the first target delay topic.
[0155] The database may be any type of database, and this application is mainly described using the rocksdb database as the database.
[0156] It can be understood that since the data is in the rocksdb database, after the rocksdb database determines that there are messages stored, it will reorder the messages based on the rocksdb compaction function and the key of the message. Since the expiration time of the message content in this application is located at the first position of the key of the message content, the message queue server will store each message content in the rocksdb database in the order from the expiration time to the back. In other words, after the message queue server receives any message content, it will reorder the stored message content based on the rocksdb compaction function according to the order of the message key from small to large, so as to store the message content in the order from the expiration time to the back.
[0157] 705: If the current time is the expiration time, the message queue server retrieves the key-value pair from the first target delay topic, and determines a first target topic corresponding to the first priority based on the first priority in the key and the first target delay topic.
[0158] Exemplarily, for the delayed topic, the message queue server will start a scheduled task for each delayed topic, and for the current moment, for the current time, it will locate the message content with the same expiration time as the current moment in the database, and read the message content from the preset database. For example, if the current moment is 1675000000, the message queue server will first locate the message content with an expiration time of 1675000000 in the database, and then take the key-value pair of the message content from the first target delayed topic, that is, take the key-value pair of the message content from the database.
[0159] Exemplarily, after the message queue server extracts the key of the message content, it will parse the key to obtain the first priority corresponding to the message content and the first target delay topic. Then, the message queue server determines the first target topic corresponding to the first priority based on the first priority and the first target delay topic.
[0160] Exemplarily, the message queue server determines the above-mentioned common topic based on the delay topic generation rule and the first target delay topic, that is, the first target delay topic is split by the delay topic generation rule to obtain the common topic. Then, the message queue server generates the first target topic based on the priority topic generation rule, the common topic, and the first priority, wherein the way the message queue server generates the first target topic is similar to the way the production-end agent generates the first target topic, which will not be described again. Among them, the delay topic generation rule and the priority topic generation rule are pre-configured to the message queue server.
[0161] 706: The message queue server publishes the message content to the first target topic.
[0162] Exemplarily, after the message queue server parses out that the message content needs to be published to the first target topic, it can select a queue from the queue of the first target topic to publish the message content to the queue, thereby completing the actual publication of the message content. It can be understood that after the message queue server publishes the message content with the delay time and priority to the first target topic, the consumer can consume the message content according to the above-mentioned message consumption process, and no further description will be given.
[0163] It can be seen that in an embodiment of the present application, for messages that require delayed publication and have priority, the present application can carry priority indication information and delay time in the message publishing request, so that the production-side agent can first publish the message to the delay topic based on the delay time, and delay the message for any time. After the delay time is over, the message will be taken out of the delay topic and distributed to the priority topic corresponding to the priority, so as to publish the message according to the priority of the message, thereby realizing both delayed publication of the message and publication of priority messages, and for the business side, only one ordinary Topic needs to be maintained to complete the delay and priority publication of the message, with low complexity and low business cost.
[0164] In conjunction with the message publishing method and message consumption method of this application, the application scenarios and related services of this application are introduced.
[0165] 1. Query the queue-jumping scenario.
[0166] Exemplarily, query requests sent by different production ends have different priorities, so that the production end agent executes the message publishing method of the present application, and publishes query requests of different priorities to priority topics corresponding to different priorities; since the consumption weight of high-priority query requests is relatively high, when pulling messages multiple times, messages are frequently pulled from the priority topics corresponding to the high priority for consumption, that is, the consumption frequency of high-priority messages is higher than that of low-priority messages, so that even if the production time of a high-priority query request is later than that of a low-priority query request, the high-priority query request may be consumed in advance, thereby realizing query queue jumping.
[0167] 2. Message push scenario.
[0168] Exemplarily, the message push scenarios include long link push scenarios and red envelope message push scenarios.
[0169] The long link push scenario refers to messages pushed through long links. For such messages, the consumer side needs to be able to consume the message in real time, that is, the real-time requirement is relatively high. Therefore, for such messages, when the producer side produces the message, it will set the priority of the message to high priority, and then publish the message to the priority topic corresponding to the high priority, so that the message can be consumed first and consume it in real time as much as possible.
[0170] The red envelope message push scenario refers to the notification message of the red envelope activity. Since the red envelope notification message has a relatively high real-time requirement, for this type of notification message, similar to the message pushed through the long link mentioned above, it is necessary to publish this type of notification message to the priority topic corresponding to the high priority, so that users can consume the notification message of the red envelope activity in real time.
[0171] 3. Live broadcast scene.
[0172] For live broadcast scenarios, the real-time requirements are relatively high. Therefore, when the message produced by the production end (for example, the live video stream) is a message under the live broadcast business scenario, the priority of the message needs to be set to the target priority, where the target priority is greater than the priority threshold, that is, it is set to a high priority. Then the message can be published to the queue corresponding to the high priority, so that the message can be consumed in real time.
[0173] 4. Order scenario.
[0174] For order scenarios, different priorities can be set for order requests from different users. Specifically, when the order server receives order requests from different users, different priorities can be set for order requests from different users based on the user type. For example, for VIP users, the order request of the VIP user is set to a high priority, and for ordinary users, the order request of the user is set to a low priority. In this way, when the consumer end (i.e., the order processing server) consumes these order requests, it can give priority to consuming the high-priority order requests, that is, give priority to consuming the order requests of VIP users, and then give priority to processing the VIP order requests, and give priority to dispatching orders to VIP users.
[0175] 5. E-commerce scenarios.
[0176] When an e-commerce company starts a promotion, it is necessary to prepare coupon messages in advance, that is, to set a delayed message, and schedule the coupon message to be published and consumed at a specific time. Moreover, when the specific time arrives, with the start of the promotion, the merchant will publish coupon messages, update inventory, etc.; and the user will place an order, generate order messages, logistics messages, and the user will generate behavior messages during the ordering process. Therefore, for these messages, coupon messages, inventory update messages, and order messages are all messages with relatively high priority, while the priority of logistics messages is second, and the priority of user behavior messages is even lower. Therefore, coupon messages, inventory update messages, and order messages can be published to priority topics corresponding to high priorities, so as to facilitate real-time consumption of coupon messages, inventory update messages, and order messages, so as to timely update the number of coupons, inventory quantity, and lock orders, etc.; publish logistics messages to priority topics corresponding to medium priorities, so as to timely update logistics information; publish user behavior messages to priority topics corresponding to low priorities, so as to facilitate later consumption of user behavior messages and analyze user behavior.
[0177] See also Fig. 9 , Fig. 9 A schematic diagram of a production-side proxy provided in an embodiment of the present application. The production-side proxy 900 includes: a transceiver unit 901 and a processing unit 902, wherein:
[0178] The transceiver unit 901 is configured to receive a first message publishing request from a production end, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic;
[0179] A processing unit 902 is configured to determine a first target topic corresponding to the first priority based on the first priority and the common topic;
[0180] The transceiver unit 901 is further used to send a second message publishing request to the message queue server to publish the message content to the first target topic, wherein the second message publishing request includes the message content and the first target topic.
[0181] In some implementations of the present application, in determining a first target topic corresponding to the first priority based on the first priority and the common topic, the processing unit 902 is specifically configured to:
[0182] If the first priority is the priority corresponding to the common topic, taking the common topic as the first target topic;
[0183] If the first priority is not the priority corresponding to the common topic, the first target topic is determined based on a pre-configured priority topic generation rule, the first priority and the common topic.
[0184] In some implementations of the present application, before determining the target topic corresponding to the first priority based on the first priority and the common topic, the processing unit 902 is further configured to:
[0185] If the first priority is greater than the maximum priority or less than the minimum priority, the minimum priority is used as the first priority.
[0186] See also Fig.10 , Fig.10 A schematic diagram of a consumer-side proxy provided in an embodiment of the present application. The consumer-side proxy 1000 includes: a transceiver unit 1001 and a processing unit 1002, wherein:
[0187] The transceiver unit 1001 is configured to receive a first message consumption request from a consumer, wherein the first message consumption request includes a common topic;
[0188] The processing unit 1002 is configured to obtain a second priority corresponding to the first message consumption request;
[0189] Based on the second priority and the general topic, determining a second target topic corresponding to the second priority;
[0190] Consume messages based on the second target topic.
[0191] In some implementations of the present application, in terms of obtaining the second priority corresponding to the first message consumption request, the processing unit 1002 is specifically configured to:
[0192] The highest priority is used as the second priority corresponding to the first message consumption request.
[0193] In some implementations of the present application, in terms of obtaining the second priority corresponding to the first message consumption request, the processing unit 1002 is specifically configured to:
[0194] Based on the preset weight corresponding to each priority, generate a random number for the first message consumption request, wherein the higher the priority, the greater the corresponding preset weight;
[0195] The priority level corresponding to the random number is used as the second priority level corresponding to the first message consumption request.
[0196] In some implementations of the present application, in determining a second target topic corresponding to the second priority based on the second priority and the common topic, the processing unit 1002 is specifically configured to:
[0197] If the second priority is the priority corresponding to the common topic, taking the common topic as the second target topic;
[0198] If the second priority is not the priority corresponding to the common topic, the second target topic is determined based on a preconfigured priority topic generation rule, the second priority and the common topic.
[0199] In some implementations of the present application, the first message consumption request includes a first quantity, wherein the first quantity is used to indicate that at most the first quantity of messages are consumed; in terms of consuming messages based on the second target topic, the processing unit 1002 is specifically configured to:
[0200] If the number of messages cached in the cache area corresponding to the second target topic is greater than or equal to the first number, pulling the first number of messages from the cache area corresponding to the second target topic;
[0201] If the number of messages cached in the cache area corresponding to the second target topic is less than the first number, all messages cached in the cache area corresponding to the second target topic are pulled, and messages are continued to be pulled from the cache area corresponding to the third target topic until the number of pulled messages reaches the first number, wherein the third target topic is a priority topic corresponding to the next priority level of the second priority;
[0202] The first number of messages is returned to the consumer.
[0203] See also Fig.11 , Fig.11 A schematic diagram of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device 1100 includes a transceiver 1101, a processor 1102 and a memory 1103. They are connected via a bus 1104. The memory 1103 is used to store computer programs and data, and can transmit the data stored in the memory 1103 to the processor 1102. Optionally, the electronic device 1100 can be the above-mentioned production-end agent 900 or the consumption-end agent 1000.
[0204] Optionally, when the electronic device 1100 is the above-mentioned production-end agent 900, the processor 1102 is used to read the computer program in the memory 1103 to perform the following operations:
[0205] Control the transceiver 1101 to receive a first message publishing request from the production end, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic;
[0206] Based on the first priority and the general topic, determining a first target topic corresponding to the first priority;
[0207] The control transceiver 1101 sends a second message publishing request to the message queue server to publish the message content to the first target topic, wherein the second message publishing request includes the message content and the first target topic.
[0208] Specifically, the control transceiver 1101 may be Fig. 9 The transceiver unit 901 of the production agent 900 of the embodiment described above, the processor 1102 may be Fig. 9 The processing unit 902 of the production-side proxy 900 of the described embodiment.
[0209] Optionally, when the electronic device 1100 is the above-mentioned consumer-end agent 1000, the processor 1102 is configured to read the computer program in the memory 1103 to perform the following operations:
[0210] Controlling the transceiver 1101 to receive a first message consumption request from a consumer end, wherein the first message consumption request includes a common topic;
[0211] Obtaining a second priority corresponding to the first message consumption request;
[0212] Based on the second priority and the general topic, determining a second target topic corresponding to the second priority;
[0213] Consume messages based on the second target topic.
[0214] Specifically, the control transceiver 1101 may be Fig.10 The transceiver unit 1001 of the consumer agent 1000 of the embodiment described above, the processor 1102 may be Fig.10 The processing unit 1002 of the consumer-side proxy 1000 of the described embodiment.
[0215] It should be noted that the message queue server, production-end agent, and consumer-end agent mentioned above can all be servers, and the server can be a cloud computing server, a content delivery network (Content Delivery Network, CDN) server, a network time protocol (Network Time Protocol, NTP), a domain name resolution system (Domain Name System, DNS) server, and other types of servers. The above servers are only examples, not exhaustive, and include but are not limited to the above servers.
[0216] The electronic devices in this application may include smart phones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, PDAs, laptop computers, mobile Internet devices MID (Mobile Internet Devices, referred to as: MID) or wearable devices, etc. The above electronic devices are only examples, not exhaustive, and include but are not limited to the above electronic devices. In practical applications, the above electronic devices may also include: smart vehicle terminals, computer equipment, etc.
[0217] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any message publishing method or message consumption method recorded in the above method embodiments.
[0218] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any message publishing method or message consumption method recorded in the above method embodiments.
[0219] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0220] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0221] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.
[0222] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0223] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.
[0224] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.
[0225] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.
[0226] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A message publishing method, characterized in that: Applied to a production-side proxy, the method includes: Receive a first message publishing request from a production end, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic; Based on the first priority and the general topic, determining a first target topic corresponding to the first priority; A second message publishing request is sent to the message queue server to publish the message content to the first target topic, wherein the second message publishing request includes the message content and the first target topic.
2. The method according to claim 1, characterized in that The determining, based on the first priority and the common topic, a first target topic corresponding to the first priority includes: If the first priority is the priority corresponding to the common topic, taking the common topic as the first target topic; If the first priority is not the priority corresponding to the common topic, the first target topic is determined based on a pre-configured priority topic generation rule, the first priority and the common topic.
3. The method according to claim 1 or 2, characterized in that: Before determining a target topic corresponding to the first priority based on the first priority and the common topic, the method further includes: If the first priority is greater than the maximum priority or less than the minimum priority, the minimum priority is used as the first priority.
4. A message consumption method, characterized in that: Applied to a consumer-side agent, the method includes: Receiving a first message consumption request from a consumer, wherein the first message consumption request includes a common topic; Obtaining a second priority corresponding to the first message consumption request; Based on the second priority and the general topic, determining a second target topic corresponding to the second priority; Consume messages based on the second target topic.
5. The method according to claim 4, characterized in that The acquiring the second priority corresponding to the first message consumption request includes: The highest priority is used as the second priority corresponding to the first message consumption request.
6. The method according to claim 4, characterized in that The acquiring the second priority corresponding to the first message consumption request includes: Based on the preset weight corresponding to each priority, generate a random number for the first message consumption request, wherein the higher the priority, the greater the corresponding preset weight; The priority level corresponding to the random number is used as the second priority level corresponding to the first message consumption request.
7. The method according to any one of claims 4 to 6, characterized in that: The determining, based on the second priority and the common topic, a second target topic corresponding to the second priority includes: If the second priority is the priority corresponding to the common topic, taking the common topic as the second target topic; If the second priority is not the priority corresponding to the common topic, the second target topic is determined based on a preconfigured priority topic generation rule, the second priority and the common topic.
8. The method according to any one of claims 4 to 7, characterized in that: The first message consumption request includes a first quantity, wherein the first quantity is used to indicate that at most the first quantity of messages are consumed; and the message consumption based on the second target topic includes: If the number of messages cached in the cache area corresponding to the second target topic is greater than or equal to the first number, pulling the first number of messages from the cache area corresponding to the second target topic; If the number of messages cached in the cache area corresponding to the second target topic is less than the first number, all messages cached in the cache area corresponding to the second target topic are pulled, and messages are continued to be pulled from the cache area corresponding to the third target topic until the number of pulled messages reaches the first number, wherein the third target topic is a priority topic corresponding to the next priority level of the second priority; The first number of messages is returned to the consumer.
9. A production-side agent, characterized in that: include: Transceiver unit and processing unit; The transceiver unit is used to receive a first message publishing request from a production end, wherein the first message publishing request includes message content, a first priority of the message content, and a common topic; The processing unit is configured to determine a first target topic corresponding to the first priority based on the first priority and the common topic; The transceiver unit is further used to send a second message publishing request to the message queue server to publish the message content to the first target topic, wherein the second message publishing request includes the message content and the first target topic.
10. A consumer-side agent, characterized in that: include: Transceiver unit and processing unit; The transceiver unit is used to receive a first message consumption request from a consumer end, wherein the first message consumption request includes a common topic; The processing unit is used to obtain a second priority corresponding to the first message consumption request; Based on the second priority and the general topic, determining a second target topic corresponding to the second priority; Consume messages based on the second target topic.
11. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 8.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 8.
