Message processing method, apparatus, device, readable storage medium and program product
By building a metadata center and dynamically adjusting the baseline consumption range using JavaAgent, the problem of swimlane message backlog was solved, ensuring the stability of the message queue system and avoiding additional modifications and costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU PINGPONG INTELLIGENT TECH CO LTD
- Filing Date
- 2026-03-19
- Publication Date
- 2026-05-29
AI Technical Summary
In a microservice architecture, swimlane messages cannot be consumed in message queue scenarios, leading to backlog issues, especially when swimlanes do not have corresponding consumer applications deployed, making it impossible to process messages with specific identifiers.
A metadata center is built to store the registration metadata of swimlane applications. Swimlane change events are pulled through the baseline environment JavaAgent to determine consumer coverage, generate a list of missing consumer identifiers, update the baseline consumption scope to achieve fallback consumption, and dynamically adjust the baseline consumer subscription rules to ensure that messages can be consumed by the baseline swimlane.
This enables timely consumption of all messages even when no consumers are deployed in non-baseline swimlanes, avoiding message backlog and maintaining the stable operation of the message queue system, all without requiring modification to the application code, thus reducing transformation costs.
Smart Images
Figure CN122120221A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a message processing method, apparatus, device, readable storage medium, and program product. Background Technology
[0002] In a microservice architecture, traffic swimlane technology is used to support scenarios such as canary releases and A / B testing. Traffic swimlane technology tags requests (i.e., swimlane identifiers), enabling specific traffic to be routed to specific microservice instances or environments in the call chain according to the swimlane identifiers on the requests and pre-defined rules. This achieves environmental isolation between different traffic types, thereby ensuring the normal operation of the production environment.
[0003] To meet specific needs such as canary releases and A / B testing within a microservices architecture, the architecture designed based on traffic swimlane technology comprises two parts: a baseline environment and a swimlane environment. The baseline environment is a complete benchmark operating environment, handling the device's normal traffic processing tasks. The swimlane environment is set up for specific testing or verification purposes, deploying only a subset of applications.
[0004] Applying swimlane schemes designed based on baseline and swimlane environments to message queue scenarios results in a situation where only consumers in specific environments can process messages with certain identifiers. When no corresponding consumer application is deployed for a swimlane, this leads to a continuous backlog of unconsumable swimlane messages. Therefore, a method is needed to resolve the issue of unconsumable swimlane message backlog in message queue scenarios. Summary of the Invention
[0005] Therefore, it is necessary to provide a message processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can solve the problem of unconsumable swimlane messages in message queue scenarios, in order to address the aforementioned technical problems.
[0006] Firstly, this application provides a message processing method, including:
[0007] A metadata center is constructed to store the registration metadata of each lane application and to determine the deployment information of each corresponding lane based on the registration metadata; the lanes include baseline lanes and non-baseline lanes.
[0008] By using the baseline environment JavaAgent to pull the swimlane change events pushed by the metadata center based on the deployment information, the consumer coverage of each non-baseline swimlane under the consumption theme is determined;
[0009] A list of lane markers for missing consumers will be determined based on the consumer coverage described above;
[0010] The baseline consumption range of the baseline swimlane is updated according to the swimlane identifier list so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
[0011] In one embodiment, updating the baseline consumption range of the baseline lane according to the lane identifier list includes:
[0012] Obtain the original SQL92 expression for the baseline swimlane;
[0013] The original SQL92 expression is updated based on the list of lane identifiers to obtain the updated SQL92 expression;
[0014] The baseline consumer of the baseline swimlane is triggered to resubscribe to the updated SQL92 expression to update the baseline consumer range of the baseline swimlane.
[0015] In one embodiment, triggering the baseline consumer to resubscribe to the updated SQL92 expression to update the baseline consumer's baseline consumption range includes:
[0016] If, based on the original SQL92 expression and the updated SQL92 expression, it is determined that the baseline consumer needs to be triggered to resubscribe to the updated SQL92 expression, then the old baseline consumer in the baseline swimlane will stop pulling new messages from RocketMQ.
[0017] Once the consumption processing of the existing consumer in the baseline consumer is completed, the corresponding consumption point is submitted to RocketMQ to create a new baseline consumer;
[0018] The new baseline consumer reloads the updated SQL92 expression to update the baseline consumer's baseline consumption range.
[0019] In one embodiment, the construction of the metadata center includes:
[0020] When the application in each swimlane starts, the JavaAgent scans and enhances the interception of swimlanes with MQ consumers to obtain the corresponding consumer's registration metadata; the registration metadata includes the application name, swimlane identifier, Topic, and ConsumerGroup associated with the consumer.
[0021] The registration metadata is reported to the registration center to obtain the metadata center.
[0022] In one embodiment, the method further includes:
[0023] Retrieve the business message sent by the message generator and send the business message to the RocketMQ message queue; wherein the business message carries a target swimlane identifier;
[0024] If the target swimlane corresponding to the target swimlane identifier does not have a consumer deployed to consume the business message, and the baseline consumption range includes the target swimlane identifier, then the business message is routed to the baseline swimlane consumer of the baseline swimlane through RocketMQ, so that the baseline swimlane consumer consumes the business message.
[0025] In one embodiment, the method further includes:
[0026] When a baseline consumer in the baseline swimlane consumes a message in the target non-baseline swimlane, a fallback consumption log is obtained.
[0027] The minimum consumption amount for each of the non-baseline lanes is determined based on the minimum consumption log.
[0028] If the target guaranteed consumption amount exceeds the preset value, an alarm will be triggered.
[0029] Secondly, this application also provides a message processing apparatus, comprising:
[0030] A construction module is used to build a metadata center, which stores the registration metadata of each lane application and determines the deployment information of each corresponding lane based on the registration metadata; the lanes include baseline lanes and non-baseline lanes;
[0031] The detection module is used to retrieve swimlane change events pushed by the metadata center based on the deployment information through the baseline environment JavaAgent, determine the consumer coverage of the consumption topic under each non-baseline swimlane, and determine the swimlane identifier list of missing consumers based on the consumer coverage of each consumer.
[0032] The fallback consumption module is used to update the baseline consumption range of the baseline swimlane according to the swimlane identifier list, so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
[0033] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0034] A metadata center is constructed to store the registration metadata of each lane application and to determine the deployment information of each corresponding lane based on the registration metadata; the lanes include baseline lanes and non-baseline lanes.
[0035] By using the baseline environment JavaAgent to pull the swimlane change events pushed by the metadata center based on the deployment information, the consumer coverage of each non-baseline swimlane under the consumption theme is determined;
[0036] A list of lane markers for missing consumers will be determined based on the consumer coverage described above;
[0037] The baseline consumption range of the baseline swimlane is updated according to the swimlane identifier list so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0039] A metadata center is constructed to store the registration metadata of each lane application and to determine the deployment information of each corresponding lane based on the registration metadata; the lanes include baseline lanes and non-baseline lanes.
[0040] By using the baseline environment JavaAgent to pull the swimlane change events pushed by the metadata center based on the deployment information, the consumer coverage of each non-baseline swimlane under the consumption theme is determined;
[0041] A list of lane markers for missing consumers will be determined based on the consumer coverage described above;
[0042] The baseline consumption range of the baseline swimlane is updated according to the swimlane identifier list so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
[0043] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0044] A metadata center is constructed to store the registration metadata of each lane application and to determine the deployment information of each corresponding lane based on the registration metadata; the lanes include baseline lanes and non-baseline lanes.
[0045] By using the baseline environment JavaAgent to pull the swimlane change events pushed by the metadata center based on the deployment information, the consumer coverage of each non-baseline swimlane under the consumption theme is determined;
[0046] A list of lane markers for missing consumers will be determined based on the consumer coverage described above;
[0047] The baseline consumption range of the baseline swimlane is updated according to the swimlane identifier list so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
[0048] The aforementioned message processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product construct a metadata center to manage application deployment information and consumers for each swimlane. A baseline environment JavaAgent subscribes to the metadata center to detect changes in each swimlane and determine the consumption deployment status of each swimlane. When a swimlane is found to be missing a consumer, the message for that swimlane is automatically included in the baseline consumption scope, achieving intelligent fallback for message consumption. This ensures that all swimlane messages are ultimately consumed, and that the swimlane environment does not need to deploy a complete consumption chain. Furthermore, this approach is based on JavaAgent, requiring no modification to the application code, thus avoiding SDK integration in actual business line transformations and reducing transformation costs. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0050] Figure 1 This is a flowchart illustrating a message processing method in one embodiment;
[0051] Figure 2 This is a flowchart illustrating the update method for an SQL92 expression in one embodiment;
[0052] Figure 3 This is a flowchart illustrating the method for sending swimlane messages and consuming fallback messages in one embodiment;
[0053] Figure 4 This is a timing interaction diagram of a message processing method in one embodiment;
[0054] Figure 5 This is a system architecture diagram of a message processing method in one embodiment;
[0055] Figure 6 This is a structural block diagram of a message processing device in one embodiment;
[0056] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0058] In message queue scenarios, after a producer sends a message with a swimlane identifier, only the consumer in the corresponding swimlane can consume it. If no consumer application is deployed for the swimlane, the message will never be consumed and will continue to accumulate. To ensure that messages can be consumed, a complete consumer chain must be deployed for each swimlane, increasing maintenance costs and resource consumption, which violates the principle of "deployment on demand." In addition, the creation and destruction of swimlane environments are dynamic, and statically configured consumption strategies cannot adapt to changes in the environment in real time.
[0059] Therefore, a message processing method is proposed to address the problem of message backlog and inability to be consumed when multiple environments are running in parallel.
[0060] In one embodiment, such as Figure 1 As shown, a message processing method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0061] Step 102: Construct a metadata center. The metadata center is used to store the registration metadata of each lane application and to determine the deployment information of each corresponding lane based on the registration metadata. The lanes include baseline lanes and non-baseline lanes.
[0062] Swimlane applications can include all business application instances participating in swimlane isolation / routing, such as swimlane producers and swimlane consumers. A swimlane producer is one that sends messages with swimlane tags, routing them to the correct Topic / Tag / queue or adding attributes. A swimlane consumer is one that filters / subscribes to isolate messages based on swimlane tags, avoiding cross-swimlane consumption.
[0063] A metadata center can be a centralized metadata center used to uniformly manage all swimlane environment information and maintain application deployment and consumer group information for each swimlane in real time. The metadata center monitors application registration behavior in each swimlane environment, collects and maintains metadata such as application instance deployment status, service name, and swimlane identifier, and dynamically generates swimlane environment information, i.e., deployment information, based on the metadata. The metadata center determines the deployment information of each corresponding swimlane based on the registration metadata. This can involve initializing a cross-environment data storage service, continuously collecting application instance metadata, and parsing the deployment topology of each swimlane.
[0064] Furthermore, the metadata center can use Redis as storage, provide a RESTful API for application registration and metadata querying, and support WebSocket push mechanisms to notify topology changes in real time.
[0065] For example, each swimlane application instance loads an Agent upon startup, scans and enhances the MQ consumer class, and registers metadata (which can be the consumer application's service name, corresponding swimlane identifier, role, topic in the message queue, and consumer group) to the metadata center. That is, when the application starts, it proactively reports metadata to the registry center, declaring its identity and runtime environment, and then sends a successful registration feedback. The application instance then periodically sends heartbeat packets to the metadata center to confirm its liveness. When an application instance goes offline, it proactively deregisters, and the metadata center synchronously deletes the consumer group association to confirm the deregistration is complete. Here, a Topic is a logical classification of messages in RocketMQ. Producers send messages to a specified Topic, and consumers subscribe to messages from that Topic. A Consumer Group is a collection of consumers. Consumers within the same consumer group jointly consume messages from a single Topic to improve the concurrency and reliability of message processing.
[0066] Step 104: By pulling swimlane change events pushed by the metadata center based on deployment information through the baseline environment JavaAgent, determine the consumer coverage of the consumption topic under each non-baseline swimlane.
[0067] The baseline environment JavaAgent can be a bytecode-enhanced agent running in the baseline environment JVM. It can be used to intercept and process message consumption-related operations without modifying the business code, and supports dynamic adjustment of consumption behavior. The JavaAgent can receive swimlane change events in real time from the metadata center, and periodically pull full data for verification. The results can be stored in a local cache to determine whether the consumer for a certain swimlane has been deployed or is available.
[0068] The baseline environment JavaAgent can collaborate with the metadata center: it receives swimlane change events pushed by the metadata center and triggers local consumer coverage analysis.
[0069] Swimlane change events can be notification messages reflecting changes in the deployment status of swimlane applications, enabling the baseline environment to be aware of consumer deployment changes in non-baseline swimlanes, such as when a swimlane has no consumer applications deployed. Swimlane change events can be generated and pushed when a swimlane application is registered or deregistered, and include fields such as change type, application name, and swimlane identifier. Consumer coverage indicates whether a swimlane has the capability to process the messages it is responsible for, serving as the basis for identifying missing consumers.
[0070] For example, by using the baseline environment JavaAgent to pull swimlane change events pushed by the metadata center based on deployment information, the consumer coverage of consumption topics under each non-baseline swimlane can be determined. This can be achieved by the JavaAgent periodically or event-drivenly retrieving change notifications from the metadata center and determining whether the corresponding swimlane is missing consumers based on the consumption topic. Furthermore, this operation can be implemented by pulling change events through long polling or by subscribing to change messages published by the metadata center through a message bus, thereby enabling real-time awareness of the missing consumer status of non-baseline swimlanes.
[0071] Step 106: Determine the list of lane markers for missing consumers based on the coverage of each consumer.
[0072] The swimlane identifier list is a set of identifiers for non-baseline swimlanes that currently have no consumer deployments. The swimlane identifier list can be determined by iterating through the consumer coverage results of all non-baseline swimlanes, filtering out swimlanes without deployed consumers, and extracting their identifiers to form the list.
[0073] Optionally, based on the coverage of each consumer, the consumer coverage of each preset topic in each swimlane is analyzed, and swimlanes without deployed consumers are filtered out to obtain the corresponding swimlane identifier list.
[0074] Step 108: Update the baseline consumption range of the baseline swimlane according to the swimlane identifier list, so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
[0075] One way to update the baseline consumption range is to inject the swimlane identifiers of missing consumers into the subscription filtering rules of the baseline consumers, so that they include messages from these swimlanes when pulling messages. The filtering rules can be expressed using SQL92 expressions.
[0076] The baseline consumption range can be configured to specify the range of messages that consumers actually subscribe to and process in the baseline swimlane. It can be used to control the types of messages that baseline consumers can pull and process, and can achieve fallback consumption of messages in missing swimlanes through dynamic updates.
[0077] The aforementioned message processing method continuously maintains the deployment metadata of each swimlane through a metadata center and generates swimlane change events when deployments change. The baseline environment JavaAgent responds to these events, analyzes consumer coverage, identifies missing consumer swimlanes, and forms a swimlane identifier list. This dynamically adjusts the subscription scope of baseline swimlane consumers, enabling them to pull and process messages that would otherwise be handled by missing consumer swimlanes. This achieves intelligent fallback for message consumption, ensuring that relevant messages are still consumed promptly even when no consumers are deployed in non-baseline swimlanes. This breaks the limitation that messages must be processed by their corresponding swimlanes, eliminates the risk of message backlog due to incomplete deployment, and maintains the stable operation of the message queue system. Furthermore, this approach is based on JavaAgent, requiring no changes to application code, avoiding SDK integration in actual business line transformations and reducing transformation costs.
[0078] In one exemplary embodiment, such as Figure 2 As shown, an update method for SQL92 expressions is provided, including the following steps:
[0079] Step 202: Obtain the raw SQL92 expression for the baseline swimlane.
[0080] The raw SQL92 expression can be a conditional expression conforming to the SQL92 standard used for message filtering in the initial configuration of the baseline consumer, and can be used to define the range of messages that the baseline swimlane can originally consume. The raw SQL92 expression can be, but is not limited to, filtering expressions based on message tags or composite conditional expressions based on message attributes.
[0081] Step 204: Update the original SQL92 expression based on the lane identifier list to obtain the updated SQL92 expression.
[0082] The updated SQL92 expression can be a new message filtering rule generated by incorporating the missing swimlane identifiers from the original SQL92 expression. For example, the original SQL92 expression is (laneId IS NULL OR laneId ='baseline'), and the updated SQL92 expression is:
[0083] (laneId IS NULL OR laneId = 'baseline' OR laneId IN (missing lane list)), where laneId is the lane identifier.
[0084] Step 206 triggers the baseline consumer of the baseline swimlane to resubscribe to the updated SQL92 expression to update the baseline consumer range of the baseline swimlane.
[0085] The baseline consumer can be a message queue client instance running in the baseline environment, responsible for pulling and processing messages under a specified topic. It can be used to perform actual message consumption tasks, and the consumption scope can be dynamically adjusted through configuration updates.
[0086] The baseline consumer triggering the baseline swimlane resubscribes to the updated SQL92 expression to update the baseline consumption range of the baseline swimlane. No application restart is required. The application receives the change notification, unsubscribes, and resubscribes according to the new message filtering rules.
[0087] Furthermore, the method of consumers subscribing to dynamic updates can achieve a smooth consumer switchover by comparing the old and new SQL92 expressions. When an update is needed, the old consumer can be gracefully stopped, and a new consumer can be created using the new SQL92 expression to continue consuming from the last offset, thus avoiding message loss.
[0088] In one exemplary embodiment, triggering the baseline consumer to resubscribe to the updated SQL92 expression to update the baseline consumer's baseline consumption range includes:
[0089] If, based on the original SQL92 expression and the updated SQL92 expression, it is determined that the baseline consumer needs to resubscribe to the updated SQL92 expression, then the old baseline consumer in the baseline swimlane stops pulling new messages from RocketMQ. If the existing consumption in the baseline consumer has been completed, the corresponding consumption point is submitted to RocketMQ to create a new baseline consumer. The new baseline consumer reloads the updated SQL92 expression to update the baseline consumption range of the baseline consumer.
[0090] The old baseline consumer can be a baseline consumer instance that was running before the SQL92 expression was updated. It is used to ensure the complete execution of currently ongoing consumption tasks and avoid message processing interruptions due to abrupt termination. The new baseline consumer can be a baseline consumer instance created after the old consumer has been cleaned up. It can be used to participate in message consumption with the updated consumption rules. The consumption position is the offset that the consumer submits to the message middleware (such as RocketMQ) after processing a batch of messages, and is used to record the consumption progress.
[0091] RocketMQ is a distributed messaging middleware that supports publish / subscribe and point-to-point messaging models. It can be used as a messaging infrastructure to handle message production and consumption in swimlane scenarios and supports server-side message filtering using SQL92 expressions.
[0092] For example, if, based on the original SQL92 expression and the updated SQL92 expression, it is determined that the baseline consumer needs to resubscribe to the updated SQL92 expression, the old baseline consumer in the baseline swimlane will pause initiating new pull requests from the RocketMQBroker, but will continue processing locally cached message batches. Once the existing consumption in the baseline consumer has completed processing, the corresponding consumption point is submitted to RocketMQ, a new consumer instance is initialized through the application container or framework, and registered to the same consumer group. The new baseline consumer reloads the updated SQL92 expression to update the baseline consumption range of the baseline consumer.
[0093] In the above embodiments, by obtaining the original SQL92 expression of the baseline swimlane, updating the original SQL92 expression according to the swimlane identifier list to obtain the updated SQL92 expression, and triggering the baseline consumer to resubscribe to the updated SQL92 expression, a reliable backup for the undeployed scenario is achieved, effectively preventing message backlog caused by the absence of consumers.
[0094] To achieve unified management of all lane environment information and real-time maintenance of application deployment and consumer group information for each lane, one embodiment constructs a metadata center, specifically a multi-environment metadata center, including:
[0095] When the application in each swimlane starts, the JavaAgent scans and enhances the interception of swimlanes with MQ consumers to obtain the corresponding consumer registration metadata. The registration metadata includes the application name associated with the consumer, swimlane identifier, Topic, and ConsumerGroup. The registration metadata is reported to the registry center to obtain the multi-environment metadata center.
[0096] The registration metadata can be a set of information describing the deployment context of a message queue consumer in a specific swimlane. It can be used to characterize whether a swimlane has the ability to consume messages on a specific topic, supporting the construction of a global deployment view in a multi-environment metadata center. In this embodiment, the registration metadata is obtained by scanning and identifying MQ consumer instances through a JavaAgent at application startup, and extracting attributes such as the associated application name, swimlane identifier, Topic, and ConsumerGroup. The registration metadata can include, but is not limited to, one or more of Kafka consumer registration metadata, RocketMQ consumer registration metadata, and RabbitMQ binding metadata. An MQ consumer can be an application component that subscribes to and processes messages on a specific topic in a message queue, and can be used to pull, parse, and process messages with specific identifiers.
[0097] For example, when each swimlane application starts, the JavaAgent scans and enhances the interception of swimlanes with MQ consumers to obtain the corresponding consumer registration metadata. This registration metadata is then reported to the registry center, resulting in a multi-environment metadata center. This reporting of registration metadata to the registry center, thus obtaining the multi-environment metadata center, can be achieved by the enhanced consumer module sending the collected registration metadata to a centralized registry center during initialization. The registry center aggregates data by swimlane dimension to form a unified multi-environment metadata center. This allows the consumer deployment status of each swimlane to be uniformly queried and monitored by external systems, providing data support for cross-swimlane consumption decisions.
[0098] It should be noted that typical metadata centers have a single function; for example, ZooKeeper and Nacos are used for service registration and on / offline management. However, the multi-environment metadata center in this embodiment collects data from two different data sources, Kubernetes and RocketMQ-Dashboard, and merges this data to resolve the conflict between testing efficiency and hardware resources in multi-branch parallel processing.
[0099] In the above embodiments, when the application in each swimlane starts, the JavaAgent scans and enhances the interception of swimlanes with MQ consumers to obtain the corresponding consumer registration metadata. The registration metadata includes the application name associated with the consumer, swimlane identifier, Topic, and ConsumerGroup. The registration metadata is reported to the registration center to obtain a multi-environment metadata center. Based on the multi-environment metadata center, it is possible to accurately determine whether there are missing consumers in non-baseline swimlanes, thereby triggering a fallback mechanism for dynamically expanding the consumption scope of the baseline swimlanes. This ensures that even if some swimlanes do not have consumers deployed, the relevant messages can still be processed in a timely manner, preventing message backlog.
[0100] In one exemplary embodiment, a method for sending swimlane messages and fallback consumption is provided, such as... Figure 3 As shown, it includes the following steps:
[0101] Step 302: Obtain the business message sent by the message generator and send the business message to the RocketMQ message queue; wherein, the business message carries the target swimlane identifier.
[0102] The target swimlane identifier can be a unique string identifier used to mark the swimlane to which a business message belongs. It can serve as the core basis for message routing and filtering, determining which swimlane environment's consumer should process the message. The target swimlane identifier can be dynamically extracted or injected into the business message by the message generator based on HTTP request headers, thread-local variables, or configuration rules.
[0103] Step 304: If the target swimlane corresponding to the target swimlane identifier does not have a consumer that consumes business messages deployed, and the baseline consumption range includes the target swimlane identifier, then the business message is routed to the baseline swimlane consumer of the baseline swimlane through RocketMQ so that the baseline swimlane consumer can consume the business message.
[0104] Among them, the baseline swimlane consumer can be a message consumer instance deployed in the baseline environment that has the ability to consume across swimlanes.
[0105] For example, obtaining the business message sent by the message producer and sending the business message to the RocketMQ message queue can be achieved by the message producer sending the constructed business message (including the target swimlane identifier) asynchronously or synchronously to the specified Topic via the RocketMQProducerAPI.
[0106] If the target swimlane corresponding to the target swimlane identifier does not have a consumer deployed to consume business messages, and the baseline consumption scope includes the target swimlane identifier, then RocketMQ will route the business message to the baseline swimlane consumer of the baseline swimlane so that the baseline swimlane consumer can consume the business message. This can be achieved by RocketMQBroker parsing the target swimlane identifier in the message attributes on the server side and matching it with the SQL92 expression registered when the baseline consumer subscribes. If the match is successful, the message will be delivered to an available instance within the baseline consumer group.
[0107] Furthermore, if the target swimlane has consumers that consume service messages, the corresponding registration metadata is registered with the metadata center. The metadata center pushes the swimlane change event of the target swimlane to the baseline environment based on the registration metadata. The baseline environment JavaAgent pulls the swimlane change event of the target swimlane and updates the baseline consumption range of the baseline swimlane on the original basis, so that the baseline swimlane can consume messages from each target non-baseline swimlane in the swimlane identifier list. That is, the SQL92 expression is recalculated, the consumer subscription is updated, and the subscription of the target swimlane is removed. Subsequently, the messages of the target swimlane can be consumed by the target swimlane itself.
[0108] In the above embodiments, by obtaining the business message sent by the message generator and sending the business message to the RocketMQ message queue, and if the target swimlane corresponding to the target swimlane identifier does not have a consumer for consuming business messages deployed and the baseline consumption range includes the target swimlane identifier, the business message is routed to the baseline swimlane consumer of the baseline swimlane through RocketMQ. RocketMQ performs real-time matching of messages on the server side based on the SQL92 expression registered by the baseline consumer. When the target swimlane identifier falls into the expanded baseline consumption range, the Broker directly delivers the message to the baseline swimlane consumer, enabling the baseline swimlane consumer to natively process messages belonging to other swimlanes without the need for an additional adaptation layer. This allows message consumption responsibility to be dynamically redistributed at runtime, completely eliminating the message backlog problem caused by the non-deployment of the target swimlane consumer.
[0109] Based on the above embodiments, a fallback consumption and monitoring system is further implemented. In an exemplary embodiment, when a baseline consumer in a baseline swimlane consumes messages targeting non-baseline swimlanes, a fallback consumption log is obtained; the fallback consumption amount for each non-baseline swimlane is determined based on the fallback consumption log; if the target fallback consumption amount exceeds a preset value, an alarm is triggered.
[0110] The fallback consumption log can be a structured operational log generated by the baseline consumer when processing target non-baseline swimlane messages. It provides observable evidence of fallback consumption behavior, supporting subsequent statistical analysis and anomaly identification. In this embodiment, the fallback consumption log is recorded by the baseline consumer after successful message consumption, using a log framework to include log entries containing fields such as target swimlane identifier, message ID, consumption timestamp, and Topic. The preset value can be a threshold configuration item used to determine whether the fallback consumption behavior is abnormal. It can be used as a criterion for triggering alarms, transforming the fallback consumption volume into a decision-making operational signal. The preset value can include, but is not limited to, one or more of the following: time window threshold, message quantity threshold, and consumption latency threshold.
[0111] For example, when a baseline consumer in a baseline swimlane consumes messages from a target non-baseline swimlane, a fallback consumption log is obtained. This can be achieved by the baseline consumer calling a log component to output a structured log containing key fields such as the target swimlane identifier after successfully executing the business logic for the target non-baseline swimlane message. The fallback consumption volume for each non-baseline swimlane is determined based on the fallback consumption log. This can be achieved by a log collection system collecting the fallback consumption log in real time, grouping and counting it according to the target swimlane identifier, and generating a consumption volume indicator at the unit time granularity. If the target fallback consumption volume exceeds a preset value, an alarm is triggered. This can be achieved by a monitoring system continuously comparing the fallback consumption volume indicator of each swimlane with the corresponding preset value. When the limit is exceeded for N consecutive periods, an alarm notification is triggered to the operations and maintenance platform or development group to achieve closed-loop repair of swimlane deployment issues.
[0112] In the above embodiments, when the baseline consumer consumes a swimlane message, a fallback consumption log is recorded. Based on the fallback consumption log, the fallback consumption volume of each swimlane is statistically analyzed. An alarm is triggered when a swimlane is continuously subjected to fallback consumption, making swimlane deployment defects no longer hidden. Operations and development teams can intervene and repair in a timely manner based on quantitative data, thereby shortening the consumer loss cycle of non-baseline swimlanes and reducing long-term reliance on the baseline fallback capability. While solving the message backlog problem, it also ensures the stability of the messaging system.
[0113] In one exemplary embodiment, such as Figure 4 As shown, a time-series interaction diagram of the above message processing method is provided, including swimlane 1 - order service (producer), multi-environment metadata center, baseline environment JavaAgent, baseline environment payment service consumer, RocketMQ Broker, and swimlane 1 - payment service (consumer).
[0114] In RocketMQ, the Broker node is responsible for storing, delivering, and retrieving messages. Swimlane 1 - the order service (generator) can be deployed on the machine / container of the business application, the payment service consumer can be deployed on the machine / container of the payment service, and the RocketMQ Broker can be deployed on a separate MQ server / container / cluster node (isolated from the business). The baseline environment JavaAgent is deployed along with each JVM application that needs swimlane capabilities (one agent is installed in each of the order service, payment service, etc. processes), and is mounted through startup parameters.
[0115] For example, based on Figure 4The interaction diagram shows that during the swimlane application startup and registration phases, swimlane 1 registers metadata with the multi-environment data center. This registration metadata can include the application name, swimlane identifier and role (which can be producer or consumer), topic, and consumer group. The multi-environment metadata center records the deployment information of swimlane 1. If swimlane 1 is found to have no consumers, the change event is pushed to the baseline environment JavaAgent, which actively pulls the change event. The baseline environment JavaAgent analyzes the consumer coverage of swimlane 1. If it detects that a consumer topic is missing in swimlane 1, a new SQL92 expression is generated based on the swimlane identifier of swimlane 1. For example, the generated new SQL92 expression could be (laneId IS NULL OR laneId = 'baseline' OR laneId='lane-1')). The baseline environment JavaAgent triggers the baseline environment payment service consumer to resubscribe to the new SQL92 expression. The baseline environment payment service consumer will pause pulling messages, wait for the messages being processed to complete, commit the offset, close the old consumer, and create a new consumer. After successful subscription, message pulling begins.
[0116] The process involves sending swimlane messages and handling fallback consumption. Specifically, swimlane 1 (the order service) sends an order message to the RocketMQ Broker. This order message carries the swimlane identifier lane-1. The RocketMQ Broker stores the order message in the consumption topic. If the payment service consumer in the baseline environment matches the corresponding SQL92 expression, the order message is processed according to business logic.
[0117] If the payment service of swimlane 1 - payment service (consumer) is deployed to swimlane 1 and the consumer metadata is registered, the multi-environment data center will push change events based on this, recalculate the SQL92 expression, update the consumer subscription to remove swimlane 1 as a fallback, and then the messages of swimlane 1 can be consumed by swimlane 1 itself.
[0118] In the above embodiments, a metadata center is constructed to manage the application deployment information and consumers of each swimlane. The baseline environment JavaAgent subscribes to the metadata center to detect changes in each swimlane and determine the consumption deployment status of each swimlane. When a swimlane is found to be missing a consumer, the message of that swimlane is automatically included in the baseline consumption scope, realizing intelligent fallback for message consumption. This ensures that all swimlane messages can eventually be consumed, and that the swimlane environment does not need to deploy a complete consumption chain. In addition, this method is implemented based on JavaAgent, and the application code does not need to be modified. In actual business line transformation, SDK integration is avoided, reducing transformation costs.
[0119] In one exemplary embodiment, such as Figure 5 As shown, a system architecture diagram of the message processing method is provided. This architecture diagram reflects the message queue fallback consumption method that supports gray-scale environments. It solves the problem of swimlane MQ messages being unable to be consumed when multiple environments are running in parallel without intrusion, application restart, or additional cost.
[0120] The system receives HTTP requests through the baseline swimlane v0, which carries a routing tag `routeTag=v0 / v1 / v2`. Traffic is then distributed to different application instances based on the routing tag. Application A - Gateway (v0) acts as the traffic entry point, routing requests to the corresponding version instance of application B - Creator (v0 / v1 / v2) based on the `routeTag` value. Each application instance then sends a message with the `routeTag` identifier to the RocketMQ message queue (TopicA). Application D capable of consuming this message is deployed in swimlane v0, but not in swimlane v1 or v2. Based on RocketMQ, a fallback subscription is implemented using SQL92 filtering conditions (`laneId='v0' OR laneId IN('v1')`). This means that the baseline consumer in swimlane v0 must consume messages from swimlane v1, while swimlane v2 can consume messages from its own swimlane.
[0121] RocketMQ collects consumer group information for TopicA from a multi-environment metadata center, and periodically retrieves deployment information for v1 and v2 swimlanes.
[0122] In the above embodiments, the metadata center continuously maintains the deployment metadata of each swimlane and generates swimlane change events when deployment changes. The baseline environment JavaAgent responds to the events, analyzes consumer coverage, identifies missing consumer swimlanes, forms a swimlane identifier list, and then dynamically adjusts the subscription scope of baseline swimlane consumers. This enables them to pull and process messages that should have been processed by the missing consumer swimlanes, achieving intelligent fallback for message consumption. This ensures that even if no consumers are deployed in non-baseline swimlanes, relevant messages can still be consumed in a timely manner, thus breaking the restriction that messages must be processed by the corresponding swimlane, eliminating the risk of message backlog caused by incomplete deployment, and maintaining the stable operation of the message queue system. In addition, this method is based on JavaAgent, requiring no changes to the application code, avoiding SDK integration in actual business line transformations, and reducing transformation costs.
[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0124] Based on the same inventive concept, this application also provides a message processing apparatus for implementing the message processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more message processing apparatus embodiments provided below can be found in the limitations of the message processing method described above, and will not be repeated here.
[0125] In one exemplary embodiment, such as Figure 6 As shown, a message processing device is provided, including: a construction module 602, a detection module 604, and a fallback consumption module 606, wherein:
[0126] Module 602 is used to build a metadata center. The metadata center stores the registration metadata of each lane application and determines the deployment information of each corresponding lane based on the registration metadata. The lanes include baseline lanes and non-baseline lanes.
[0127] The detection module 604 is used to pull swimlane change events pushed by the metadata center based on deployment information through the baseline environment JavaAgent to determine the consumer coverage of the consumption topic under each non-baseline swimlane; and to determine the swimlane identifier list of missing consumers based on the consumer coverage of each consumer.
[0128] The fallback consumption module 606 is used to update the baseline consumption range of the baseline swimlane according to the swimlane identifier list, so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
[0129] The aforementioned message processing device manages application deployment information and consumers for each swimlane by constructing a metadata center. A baseline environment JavaAgent subscribes to the metadata center to detect changes in each swimlane and determine its consumption deployment status. When a swimlane is found to be missing a consumer, it automatically includes the swimlane's messages in the baseline consumption scope, achieving intelligent fallback for message consumption. This ensures that all swimlane messages are ultimately consumed, and that the swimlane environment does not need to deploy a complete consumption chain. Furthermore, this approach is based on JavaAgent, requiring no changes to the application code, thus avoiding SDK integration during actual business line transformation and reducing transformation costs.
[0130] In one exemplary embodiment, the message processing apparatus further includes an SQL92 expression generation module, which is used to obtain the original SQL92 expression of the baseline swimlane; and update the original SQL92 expression according to the swimlane identifier list to obtain the updated SQL92 expression.
[0131] The update module is used to trigger the baseline consumer of the baseline swimlane to resubscribe to the updated SQL92 expression to update the baseline consumer range of the baseline swimlane.
[0132] In an exemplary embodiment, the update module is configured to, if based on the original SQL92 expression and the updated SQL92 expression, determine that the baseline consumer needs to be triggered to resubscribe to the updated SQL92 expression, then the old baseline consumer in the baseline swimlane stops pulling new messages from RocketMQ.
[0133] Once the consumption processing of an existing consumer in the baseline consumer is completed, the corresponding consumption point is submitted to RocketMQ to create a new baseline consumer;
[0134] The new baseline consumer reloads the updated SQL92 expression to update the baseline consumer's baseline consumption range.
[0135] In an exemplary embodiment, the construction module 602 is used to scan and perform enhanced interception on swimlanes with MQ consumers by JavaAgent when the swimlane application starts, and obtain the corresponding consumer registration metadata; the registration metadata includes the application name associated with the consumer, swimlane identifier, Topic and ConsumerGroup; and report the registration metadata to the registration center to obtain the metadata center.
[0136] In an exemplary embodiment, the fallback consumption module 606 is used to obtain the business message sent by the message generator and send the business message to the RocketMQ message queue; wherein the business message carries a target swimlane identifier;
[0137] If the target swimlane corresponding to the target swimlane identifier does not have a consumer that consumes business messages deployed, and the baseline consumption scope includes the target swimlane identifier, then the business messages will be routed to the baseline swimlane consumer of the baseline swimlane via RocketMQ, so that the baseline swimlane consumer can consume the business messages.
[0138] In an exemplary embodiment, the message processing device further includes an alarm module, which is used to obtain a fallback consumption log when a baseline consumer in a baseline swimlane consumes messages targeting a non-baseline swimlane; determine the fallback consumption amount for each non-baseline swimlane based on the fallback consumption log; and issue an alarm if the target fallback consumption amount is greater than a preset value.
[0139] Each module in the aforementioned message processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.
[0140] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a device bus, and the communication interface, display unit, and input device are connected to the device bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores operating devices and computer programs. The internal memory provides an environment for the operation of the operating devices and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a message processing method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0141] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0142] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0144] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0145] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A message processing method, characterized in that, The method includes: A metadata center is constructed to store the registration metadata of each lane application and to determine the deployment information of each corresponding lane based on the registration metadata; the lanes include baseline lanes and non-baseline lanes. By using the baseline environment JavaAgent to pull the swimlane change events pushed by the metadata center based on the deployment information, the consumer coverage of each non-baseline swimlane under the consumption theme is determined; A list of lane markers for missing consumers will be determined based on the consumer coverage described above; The baseline consumption range of the baseline swimlane is updated according to the swimlane identifier list so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
2. The method according to claim 1, characterized in that, The step of updating the baseline consumption range of the baseline lane according to the lane identifier list includes: Obtain the original SQL92 expression for the baseline swimlane; The original SQL92 expression is updated based on the list of lane identifiers to obtain the updated SQL92 expression; The baseline consumer of the baseline swimlane is triggered to resubscribe to the updated SQL92 expression to update the baseline consumer range of the baseline swimlane.
3. The method according to claim 2, characterized in that, The step of triggering the baseline consumer to resubscribe to the updated SQL92 expression to update the baseline consumer's baseline consumption range includes: If, based on the original SQL92 expression and the updated SQL92 expression, it is determined that the baseline consumer needs to be triggered to resubscribe to the updated SQL92 expression, then the old baseline consumer in the baseline swimlane will stop pulling new messages from RocketMQ. Once the consumption processing of the existing consumer in the baseline consumer is completed, the corresponding consumption point is submitted to RocketMQ to create a new baseline consumer; The new baseline consumer reloads the updated SQL92 expression to update the baseline consumer's baseline consumption range.
4. The method according to claim 1, characterized in that, The construction of the metadata center includes: When the application in each swimlane starts, the JavaAgent scans and enhances the interception of swimlanes with MQ consumers to obtain the corresponding consumer's registration metadata; the registration metadata includes the application name, swimlane identifier, Topic, and ConsumerGroup associated with the consumer. The registration metadata is reported to the registration center to obtain the metadata center.
5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Retrieve the business message sent by the message generator and send the business message to the RocketMQ message queue; wherein the business message carries a target swimlane identifier; If the target swimlane corresponding to the target swimlane identifier does not have a consumer deployed to consume the business message, and the baseline consumption range includes the target swimlane identifier, then the business message is routed to the baseline swimlane consumer of the baseline swimlane through RocketMQ, so that the baseline swimlane consumer consumes the business message.
6. The method according to claim 5, characterized in that, The method further includes: When a baseline consumer in the baseline swimlane consumes a message in the target non-baseline swimlane, a fallback consumption log is obtained. The minimum consumption amount for each of the non-baseline lanes is determined based on the minimum consumption log. If the target guaranteed consumption amount exceeds the preset value, an alarm will be triggered.
7. A message processing device, characterized in that, The device includes: A construction module is used to build a metadata center, which stores the registration metadata of each lane application and determines the deployment information of each corresponding lane based on the registration metadata; the lanes include baseline lanes and non-baseline lanes; The detection module is used to retrieve swimlane change events pushed by the metadata center based on the deployment information through the baseline environment JavaAgent, determine the consumer coverage of the consumption topic under each non-baseline swimlane, and determine the swimlane identifier list of missing consumers based on the consumer coverage of each consumer. The fallback consumption module is used to update the baseline consumption range of the baseline swimlane according to the swimlane identifier list, so that the baseline swimlane consumes messages from each target non-baseline swimlane in the swimlane identifier list.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.