Gray release method and device based on cloud application message queue
By introducing a canary deployment coordinator and queue allocation strategy on the broker server, canary deployment of message queues is realized, solving the problems of resource waste and poor user experience in existing technologies and providing an efficient canary deployment solution.
Patent Information
- Application Number
- CN202511854334.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-01-20
AI Technical Summary
Existing message queues typically require middleware modifications during canary deployments, leading to poor user experience and resource waste. They also present issues such as queue redistribution and downgraded consumption when canary consumers are taken offline.
By introducing a canary deployment coordination manager on the broker server, dividing the canary queue set and the baseline queue set, and combining the message subscription polling algorithm and the delivery algorithm, the isolation between producers and consumers is achieved, avoiding the need to apply for additional topics or machines, and supporting canary deployment.
It achieves production and consumption isolation between baseline and canary release, improves user experience, avoids resource waste, and supports seamless upgrade and downgrade functions during the canary release process.
Smart Images
Figure CN121367733A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, in particular to a gray release method and device based on cloud application message queue. BACKGROUND
[0002] In the current software industry, in order to improve production quality, gray release is more and more important. At present, message queue (MQ) is usually used for gray release. However, there is no message queue that actively supports gray release. Usually, the basic functions of the message middleware are simply modified to meet the basic needs of gray release, but the user experience and middleware performance are sacrificed, and machine resources are wasted. In addition, gray release will bring new problems: such as queue reallocation problem under the scene of node increase and decrease, gray message degradation consumption problem when gray consumer is offline, etc. SUMMARY
[0003] Therefore, the present application provides a gray release method and device based on cloud application message queue to solve the above technical problems.
[0004] In a first aspect, the present application provides a gray release method based on cloud application message queue, applied to a broker server, comprising: obtaining an online consumer at a current time; determining a topic to which the online consumer belongs by using a pre-established gray coordination manager; determining whether the topic belongs to a gray node, if yes and the topic is in a survival state, determining that the consumer is a gray consumer and a target queue is a gray queue set of the topic, otherwise, determining that the consumer is a baseline consumer and the target queue is a baseline queue set of the topic; polling the target queue by using a message subscription polling algorithm, and allocating a message queue for the online consumer.
[0005] In a possible implementation, the method further comprises: creating a plurality of message queues for each topic; grouping the message queues of all topics according to the broker server; for the plurality of message queues of each group, dividing the plurality of message queues into a baseline queue set and a gray queue set by using a queue allocation mapping table.
[0006] In a possible implementation, the method further comprises: obtaining an entry request and determining a type of a message, the type of the message being a baseline message or a gray message; determining a target message queue set according to the type of the message; the target message queue set is a baseline queue set or a gray queue set; delivering the message into a target queue in the target message queue set based on a message delivery polling algorithm.
[0007] In one possible implementation, the method further includes: determining whether the broker server is a gray server, and if yes, determining the message in the entry request as a gray message; otherwise, determining whether the entry request is a gray request, and if yes, determining the message as a gray message; otherwise, determining whether the message is gray traffic, and if yes, determining the message as a gray message.
[0008] In one possible implementation, the method further includes: sorting the queues in the target message queue set; obtaining a sequence number n of a last delivery queue, and if n=0, delivering the message to a first queue and marking the sequence number of the last delivery queue as 1; otherwise, delivering the message to an n+1th queue and marking the sequence number of the last delivery queue as n+1.
[0009] In one possible implementation, the method further includes: generating a gray name of the gray consumer based on a gray identifier, an IP address and a registration time of the gray consumer; issuing the gray name to the gray consumer.
[0010] In one possible implementation, the gray coordination manager records gray consumer information and baseline consumer information of each topic; the gray consumer information includes a gray identifier, an IP address, a registration time and historical access information. In one possible implementation, the method further includes: obtaining a name of the online consumer, and parsing the name of the online consumer to obtain a gray identifier, an IP address and a registration time of the online consumer; determining the topic to which the online consumer belongs by using the gray coordination manager based on the gray identifier, the IP address and the registration time of the online consumer.
[0011] In one possible implementation, the method further includes: When the online gray consumer is offline, the message queue assigned to it is assigned to the gray consumer with the same topic; When all online gray consumers are offline, the baseline consumer is allowed to access the queues of the gray queue set.
[0012] In a second aspect, the embodiments of the present application provide a gray release device based on a cloud application message queue, applied to a broker server, comprising: An acquisition unit is configured to acquire a consumer that is online at a current time; A first determination unit is configured to determine a topic to which the online consumer belongs by using a gray coordination manager established in advance; A second determination unit is configured to determine whether the topic belongs to a gray node, and if yes and the topic is in a live state, determine that the consumer is a gray consumer and a target queue is a gray queue set of the topic, otherwise, determine that the consumer is a baseline consumer and the target queue is a baseline queue set of the topic; A gray release unit is configured to poll the target queue by using a message subscription polling algorithm, and assign a message queue to the online consumer.
[0013] In a third aspect, the embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the embodiments of the present application.
[0014] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the method of the embodiments of the present application.
[0015] In a fifth aspect, the embodiments of the present application provide a computer program product, comprising computer program / instructions, which are executed by a processor to implement the method of the embodiments of the present application.
[0016] The embodiments of the present application use the gray queue set and the baseline queue set to implement message production isolation and consumption isolation, which is convenient to use, does not need to apply for redundant topics or machines, effectively avoids wasting resources, and the gray consumer does not need to consume a large number of baseline messages. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments or the prior art. Obviously, the drawings described in the following description are some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative work based on these drawings are within the scope of protection of the present application.
[0018] Figure 1 The architecture diagram of the gray release method provided by the embodiments of the present application is shown in the figure. Figure 2 The flowchart of the gray release method based on the cloud application message queue provided by the embodiments of the present application is shown in the figure. Figure 3 The schematic diagram of the producer delivering messages to the queue provided by the embodiments of the present application is shown in the figure. Figure 4 The schematic diagram of the consumer obtaining messages from the queue provided by the embodiments of the present application is shown in the figure. Figure 5 The schematic diagram of the gray coordination manager provided by the embodiments of the present application is shown in the figure. Figure 6 The schematic diagram of one gray consumer offline provided by the embodiments of the present application is shown in the figure. Figure 7 The schematic diagram of all gray consumers offline provided by the embodiments of the present application is shown in the figure. Figure 8 The functional structure diagram of the gray release device based on the cloud application message queue provided by the embodiments of the present application is shown in the figure. Figure 9 The structural diagram of the electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.
[0020] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.
[0021] First, the design idea of the embodiments of the present application is briefly introduced.
[0022] Taking RocketMQ as an example, the current industry conventional gray scheme has the following several kinds.
[0023] The shadow topic scheme. The core principle of this scheme is that when creating a topic, an additional gray topic name is applied, which is based on the original topic and adds a prefix or suffix. Let the gray consumer subscribe to the gray topic, and the baseline consumer only subscribes to the original baseline topic. The shadow topic needs to apply for an additional topic, and the broker needs to be modified; the user experience is poor, the message volume is doubled, and resources are wasted.
[0024] User property filtering scheme. RocketMQ's message body has a property set that can be added with data for gray tagging. The consumer verifies whether the tag matches when consuming messages, filtering out messages that do not match. Messages are sent to all nodes, and for gray nodes, they need to accept the full amount of messages and then filter them locally, which puts a lot of pressure on gray nodes, causing message accumulation.
[0025] Tag filtering scheme. RocketMQ's system attribute tag attribute, the consumer can filter when subscribing to messages, or filter after pulling messages. The Tag filtering scheme is not flexible because the Tag value can only have one value, and cannot meet the Tag usage needs of complex scenarios.
[0026] Therefore, the present application provides a gray publishing method based on a cloud application message queue, as shown in Figure 1 The core idea is to divide the gray queue set and the baseline queue set, and combine the gray coordination manager, custom message publishing algorithm, and custom message subscription algorithm to achieve message publishing and subscription isolation. The goal of the MQ gray scheme is to achieve baseline and gray production isolation and consumption isolation, as shown in the following figure, that is, gray messages are only consumed by gray nodes, and baseline messages are only consumed by baseline nodes. The present application uses the gray queue set and the baseline queue set to achieve message production isolation and consumption isolation, which is convenient to use and does not require the application of additional topics or machines, which can avoid wasting resources; in addition, the producer and the consumer have sensing and querying capabilities for the online and offline of the gray nodes, thereby realizing the gray degradation function.
[0027] After introducing the application scenarios and design ideas of the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be described below.
[0028] As shown in Figure 2 The present application provides a gray publishing method based on a cloud application message queue, applied to a broker server, comprising the following steps: Step 101: Obtain the consumers online at the current time; Step 102: determining the topic to which the online consumer belongs by using the pre-established gray-scale coordination manager; Step 103: determining whether the topic to which the consumer belongs is a gray-scale node, if yes and the topic to which the consumer belongs is in a survival state, then determining that the consumer is a gray-scale consumer and the target queue is a gray-scale queue set of the topic to which the consumer belongs, otherwise, determining that the consumer is a baseline consumer and the target queue is a baseline queue set of the topic to which the consumer belongs; Step 104: polling the target queue by using a message subscription polling algorithm and assigning a message queue to the online consumer.
[0029] The application realizes production isolation and consumption isolation of messages by using the gray-scale queue set and the baseline queue set, is convenient to use, does not need to apply for redundant topics or machines, can avoid wasting resources, and the gray-scale consumer does not need to consume a large number of baseline messages.
[0030] In some embodiments, the method further comprises: creating multiple message queues for each topic; grouping the message queues of all topics according to the broker servers; for the multiple message queues of each group, dividing the multiple message queues into a baseline queue set and a gray-scale queue set by using a queue allocation mapping table.
[0031] Specifically, a broker cluster includes multiple broker servers (broker nodes), which is a basic service node in a distributed system, and its core responsibility is to receive, store and forward messages / data. It acts as an intermediary between message producers and consumers. For each topic, the number of queues is fixed without scaling. Using a queue allocation mapping table, the queues are logically marked into two sets: a baseline queue set and a gray-scale queue set. For example, when the total number of queues is 4, the first queue will be marked in the gray-scale queue set, and the other queues will be marked in the baseline queue set.
[0032] At the same time, in order to avoid too many gray-scale queues and waste resources, when the number of queues is greater than or equal to 16, the gray-scale queue set is fixed to the three queues with serial numbers 1, 5 and 9. The queue allocation mapping table is shown in Table 1: Table 1
[0033] The application introduces a queue allocation mapping table, which solves the scaling problem of the gray-scale queue set.
[0034] The producer accesses the queue to deliver messages, for exampleFigure 3 As shown, a consumer accesses a queue to obtain a message such as Figure 4 As shown.
[0035] In some embodiments, the method further comprises: obtaining an entry request and determining a type of the message, the type of the message being a baseline message or a gray message; determining a target message queue set according to the type of the message, the target message queue set being a baseline queue set or a gray queue set; delivering the message to a target queue in the target message queue set based on a message delivery polling algorithm.
[0036] Specifically, a message subscription polling algorithm is used to poll the target queues to determine which queue to subscribe to. This strategy allocates message queues evenly to each message consumer. Assuming that a target queue has 8 message queues and there are 3 message consumers A, B, and C, then using this load strategy, A is allocated 3 message queues first, then B is also allocated 3 message queues, and finally C is allocated 2 message queues.
[0037] In some embodiments, obtaining an entry request and determining a type of the message comprises: determining whether the broker server is a gray server, and if so, determining the message in the entry request as a gray message; otherwise, determining whether the entry request is a gray request, and if so, determining the message as a gray message; otherwise, determining whether the message is gray traffic, and if so, determining the message as a gray message.
[0038] In some embodiments, delivering the message to a target queue in the target message queue set based on a message delivery polling algorithm comprises: sorting the queues in the target message queue set; obtaining a sequence number n of the last delivery queue, and if n = 0, delivering the message to the first queue and marking the sequence number of the last delivery queue as 1; otherwise, delivering the message to the n+1 queue and marking the sequence number of the last delivery queue as n+1.
[0039] In some embodiments, the method further comprises: generating a gray name of the gray consumer based on a gray identifier, an IP address, and a registration time of each gray consumer; issuing the gray name to the gray consumer.
[0040] In some embodiments, as shown in Figure 5As shown, the gray scale coordination manager records the gray scale consumer information and baseline consumer information of each topic; the gray scale consumer information includes gray scale identification, IP address, registration time and historical access information; By using the pre-established gray scale coordination manager, the topic to which the online consumer belongs is determined, including: The name of the online consumer is obtained, and the name of the consumer is parsed to obtain the gray scale identification, IP address and registration time of the online consumer; Based on the gray scale identification, IP address and registration time of the online consumer, the gray scale coordination manager is used to determine the topic to which the online consumer belongs.
[0041] In this embodiment, the core function of the gray scale coordination manager is to count the survival state of the gray scale consumer under the current system state, and a query channel is provided, each consumer can interact with the gray scale coordination manager in real time, so as to obtain the gray scale consumer information under the topic to which the consumer belongs, and can compare the historical existing gray scale consumer information with the current survival gray scale consumer information, so as to determine which gray scale consumers are offline under the current state.
[0042] In some embodiments, the method further comprises: When the online gray scale consumer is offline, the message queue allocated to it is allocated to the gray scale consumer with the same topic; as Figure 6 shown; When all online gray scale consumers are offline, the baseline consumer is allowed to access the queue of the gray scale queue set. As Figure 7 shown.
[0043] The embodiment can solve the problem of gray scale message degradation consumption when the gray scale consumer is offline.
[0044] In addition, the method further comprises: barrier-free upgrading of gray scale, which means upgrading only the rocket version without changing the producer and consumer, and the original message delivery and subscription logic remains normal.
[0045] Only the producer is upgraded to gray scale, the gray scale queue set has no data, and all messages are delivered to the baseline queue combination, and the consumer normally consumes the baseline queue message; Only the consumer is upgraded to gray scale, triggering gray scale degradation consumption, and the baseline consumer consumes all queue messages.
[0046] The embodiment can solve the technical problem of high upgrade cost that the producer and consumer must be switched at the same time when the gray scale is online without considering downward compatibility in the switching process of new and old clusters by barrier-free upgrading of gray scale.
[0047] Based on the same inventive concept, the embodiment of the present application provides a gray release device based on a cloud application message queue. Referring to Figure 8 As shown in the figure, the gray release device 200 based on the cloud application message queue provided by the embodiment of the present application at least includes: The acquisition unit 201 is configured to acquire a consumer online at a current time point. The first determination unit 202 is configured to determine a topic to which the online consumer belongs by using a pre-established gray coordination manager. The second determination unit 203 is configured to determine whether the topic belongs to a gray node. If yes and the topic is in a survival state, it is determined that the consumer is a gray consumer and a target queue is a gray queue set of the topic. Otherwise, it is determined that the consumer is a baseline consumer and the target queue is a baseline queue set of the topic. The gray release unit 204 is configured to poll the target queue by using a message subscription polling algorithm and allocate a message queue for the online consumer.
[0048] It should be noted that the principle of solving the technical problem of the gray release device 200 based on the cloud application message queue provided by the embodiment of the present application is similar to the method provided by the embodiment of the present application. Therefore, the implementation of the gray release device 200 based on the cloud application message queue provided by the embodiment of the present application can be referred to the implementation of the method provided by the embodiment of the present application, and the repeated parts will not be described herein.
[0049] Based on the same inventive concept, the embodiment of the present application further provides an electronic device, as shown in the figure, comprising a memory and a processor. The memory stores an executable program. The processor executes the executable program to implement the steps of the gray release method based on the cloud application message queue provided by the above embodiment. Figure 9
[0050] The processor can be a general processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general processor can be a microprocessor or any conventional processor.
[0051] Since the electronic device introduced in the embodiments of the present application is the electronic device provided with the memory implementing the gray release method based on the cloud application message queue disclosed in the embodiments of the present application, the structure of the electronic device introduced in the embodiments of the present application and the deformation can be understood by those skilled in the art based on the gray release method based on the cloud application message queue introduced in the embodiments of the present application, and thus will not be described here.
[0052] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the gray release method based on the cloud application message queue provided in the above embodiments are implemented.
[0053] The storage medium in the embodiments can be included in the electronic device, or can exist separately and not be assembled into the electronic device. The storage medium carries one or more computer programs, which, when executed, implement the steps of the gray release method based on the cloud application message queue provided in the above embodiments.
[0054] It should be understood that each scheme in the embodiments has the corresponding technical effects in the above method embodiments, which will not be described here.
[0055] According to the embodiments of the present application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. Alternatively, the specific examples in the embodiments can refer to the examples described in any embodiment of the present application, which will not be described here. Obviously, those skilled in the art should understand that the above modules or steps of the present application can be realized by a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in a different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
[0056] The embodiment of the present application further provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the cloud application message queue-based gray release method provided by the above embodiment.
[0057] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.
[0058] Further, although the operations of some of the embodiments are depicted in a particular, sequential order, this should not be understood as a requirement that such operations be performed in the particular order shown or in sequential order, at all. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while some of the embodiments described above are presented in the context of a single embodiment, it is understood that a combination of one or more features of one embodiment can be combined with one or more features of another embodiment. Conversely, various features of the different embodiments described above can be combined in a single embodiment.
Claims
1. A gray release method based on a cloud application message queue, applied to a broker server, and characterized in that, The method comprises: acquiring a consumer who is online at a current time; determining a topic to which the online consumer belongs by using a pre-established gray-scale coordination manager; judging whether the topic belongs to a gray-scale node, and if yes and the topic is in a survival state, determining that the consumer is a gray-scale consumer and a target queue is a gray-scale queue set of the topic, otherwise, determining that the consumer is a baseline consumer and the target queue is a baseline queue set of the topic; adopting a message subscription polling algorithm to poll the target queue and allocate a message queue for the online consumer.
2. The method of claim 1, wherein, The method further comprises: creating multiple message queues for each topic; grouping the message queues of all topics according to broker servers; for multiple message queues of each group, dividing the multiple message queues into a baseline queue set and a gray-scale queue set by using a queue allocation mapping table.
3. The method of claim 2, wherein, The method further comprises: acquiring an entry request and determining a type of a message, the type of the message being a baseline message or a gray-scale message; determining a target message queue set according to the type of the message, the target message queue set being a baseline queue set or a gray-scale queue set; delivering the message into a target queue in the target message queue set based on a message delivery polling algorithm.
4. The method of claim 3, wherein, acquiring an entry request and determining a type of a message, the type of the message being a baseline message or a gray-scale message; determining a target message queue set according to the type of the message, the target message queue set being a baseline queue set or a gray-scale queue set; delivering the message into a target queue in the target message queue set based on a message delivery polling algorithm. sorting the queues of the target message queue set; 5. The method of claim 3, wherein, acquiring a serial number n of a last delivery queue, if n=0, delivering the message into a first queue and marking the serial number of the last delivery queue as 1, otherwise, delivering the message into an n+1th queue and marking the serial number of the last delivery queue as n+1. The method further comprises: generating a gray-scale name of each gray-scale consumer based on a gray-scale identifier, an IP address and a registration time of the gray-scale consumer; 6. The method of claim 1, wherein, downloading the gray-scale name to the gray-scale consumer. The gray-scale coordination manager records gray-scale consumer information and baseline consumer information of each topic, the gray-scale consumer information including a gray-scale identifier, an IP address, a registration time and historical access information; determining a topic to which the online consumer belongs by using a pre-established gray-scale coordination manager, comprising:
7. The method of claim 6, wherein, acquiring a name of an online consumer, analyzing the name of the consumer to obtain a gray-scale identifier, an IP address and a registration time of the online consumer; determining the topic to which the online consumer belongs by using the gray-scale coordination manager based on the gray-scale identifier, the IP address and the registration time of the online consumer. The method further comprises: 8. The method of claim 1, wherein, When the online gray consumer is offline, the message queue assigned to it is assigned to the gray consumer with the same topic; When all online gray consumers are offline, the baseline consumer is allowed to access the queues of the gray queue set. 9.A gray release device based on cloud application message queue, applied to a broker server, and characterized in that, The method comprises the steps of: an acquisition unit, configured to acquire a consumer that is online at a current time; a first determination unit, configured to determine a topic to which the online consumer belongs by using a gray coordination manager established in advance; a second determination unit, configured to determine whether the topic belongs to a gray node, and if yes and the topic is in a live state, determine that the consumer is a gray consumer and a target queue is a gray queue set of the topic, otherwise, determine that the consumer is a baseline consumer and the target queue is a baseline queue set of the topic; a gray release unit, configured to poll the target queue by using a message subscription polling algorithm, and assign a message queue to the online consumer.
10. An electronic device, comprising: The method comprises the steps of: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1-8.