Data processing method and related equipment

Abstract

The invention provides a data processing method which is applied to a data processing system, the data processing system comprises a first data center, a first message queue cluster and a first consumer application program are deployed in the first data center, and an agent module is deployed in the first consumer application program. The method comprises the steps that an agent module obtains a first consumer instance of a first data center; the agent module obtains a first theme subscribed by the first consumer instance; the agent module packages the first consumer instance to generate a second consumer instance; and in response to the failure of the first data center, the agent module determines a first theme subscribed by the first consumer instance according to the second consumer instance, and cancels subscription of at least one target theme in the first theme. According to the method, on the basis that original service codes are not invasive, message queue flow control is achieved, and decoupling of service logic and message queue flow control logic is achieved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a data processing method, apparatus, computing device, computing device cluster, computer-readable storage medium, and computer program product. Background Technology

[0002] With the continuous development of cloud computing technology, more and more users (such as organizations and enterprises) are adopting microservices architecture to deploy business applications. In microservices architecture, business applications are decomposed into multiple small, independent services, each implementing specific business functions, giving business applications advantages such as rapid iteration, high availability, and scalability.

[0003] Some business applications based on a microservices architecture can be deployed in data centers, utilizing computing resources such as servers or virtual machines provided by the data center. Typically, the data center provides producer applications, consumer applications, and message queue clusters. Producer applications create producer instances, which produce messages from the business applications to the message queue cluster. Consumer applications create consumer instances, which consume messages from the business applications from the message queue cluster. Communication between multiple services within the business applications is achieved through the producer applications, consumer applications, and message queue clusters.

[0004] During the operation of business applications, if a data center failure occurs, to ensure the stability of business data, it is necessary to control the message queue traffic of consumer instances. This could involve controlling consumer instances to stop consuming at least a portion of the messages from the business applications within the message queue cluster. Typically, the industry achieves this control by modifying the business code. However, this method is somewhat intrusive to the business code and incurs significant modification costs. Summary of the Invention

[0005] This application provides a data processing method that enables message queue flow control without intruding on existing business logic. This application also provides a corresponding data processing apparatus, computing device, computing device cluster, computer-readable storage medium, and computer program product.

[0006] Firstly, this application provides a data processing method that can be executed by a data processing system. The data processing system includes a first data center, on which a first message queue cluster and a first consumer application are deployed, and an agent module is deployed within the first consumer application.

[0007] Specifically, in response to a consumer creation operation triggered by a first consumer application, the agent module obtains a first consumer instance in a first data center. In response to a topic subscription operation triggered by the first consumer application, the agent module obtains a first topic subscribed to by the first consumer instance. Then, the agent module encapsulates the first consumer instance to generate a second consumer instance. The second consumer instance includes information indicating the first consumer instance and information indicating the subscription relationship between the first consumer instance and the first topic. In response to a failure in the first data center, the agent module, based on the second consumer instance, determines the first topic subscribed to by the first consumer instance and unsubscribes from at least one target topic within the first topic, causing the first consumer instance to stop consuming messages associated with at least one target topic in the first message queue cluster.

[0008] This method deploys an agent module in the consumer application. The agent module intercepts the consumer creation operation, encapsulates the created consumer instance, and stores the topics subscribed to by the consumer instance. Thus, when a data center fails, the agent module can modify the subscription relationship through the encapsulated consumer instance, reducing message consumption by the consumer instance in the failed data center. This achieves message queue flow control without intruding on the original business code, decoupling business logic from message queue flow control logic.

[0009] In some possible implementations, the data processing system also includes a second data center, which deploys a second consumer application and a second message queue cluster. The second message queue cluster belongs to the same message queue cluster as the first message queue cluster. The third consumer instance created by the second consumer application belongs to the same consumer group as the first consumer instance. The agent module can determine the first topic subscribed to by the first consumer instance based on the second consumer instance and unsubscribe from all first topics. The agent module can also trigger a rebalancing operation for the message queues, ensuring that all messages associated with the first topic in the message queue cluster are consumed by the third consumer instance.

[0010] This method can be applied to active-active scenarios within the same city. By unsubscribing from all primary topics through the agent module and triggering a rebalancing operation, the third consumer instance can use the message queue originally used by the primary consumer instance for message consumption, thereby improving resource utilization, ensuring business continuity of business applications, and ensuring that message queue traffic in the primary data center is cleared. In disaster recovery drills, this method can achieve the integrity of the disaster recovery drill.

[0011] In some possible implementations, the data processing system also includes an access layer that, in response to a failure in the first data center, distributes received external requests to a second data center.

[0012] In this method, in a dual-active scenario within the same city, the second data center performs the message production process through request distribution at the access layer, thereby clearing the access layer traffic in the first data center.

[0013] In some possible implementations, in response to a failure in the first data center, the agent module obtains a set of topics, which includes at least one target topic that needs to be unsubscribed from. Then, based on the second consumer instance, the agent module determines the first topic subscribed to by the first consumer instance, queries the first topic subscribed to by the first consumer instance based on the topic set, and unsubscribes from at least one target topic in the first topic that matches the topic set.

[0014] In this method, by providing a set of topics, the agent module can modify the subscription relationship for some topics in the first topic, refine the granularity of message queue traffic control to the topic level, achieve more granular message queue traffic control, and improve applicability.

[0015] In some possible implementations, the agent module can obtain the subscription status of the first topic. In response to the existence of unsubscribed topics among other topics in the first topic, the agent module can subscribe to the unsubscribed topics. These other topics include topics in the first topic other than at least one target topic, and the subscription status of the unsubscribed topics is "unsubscribed."

[0016] In this method, after implementing flow control for the message queue, the subscription relationship of the first consumer instance can also be checked to ensure that the first consumer instance can normally consume messages using the uncontrolled topic, thus ensuring business continuity.

[0017] In some possible implementations, the agent module determines the original class of the first consumer instance, where the original class is represented in bytecode. Then, the agent module encapsulates the original class of the first consumer instance, creates a proxy class, and adds the first topic subscribed to by the first consumer instance to the proxy class. Based on the proxy class, the agent module generates the second consumer instance.

[0018] In this method, the agent module implements instance encapsulation based on bytecode enhancement technology. During the encapsulation process, the agent module can add additional functional logic to the proxy class, such as adding methods for subscribing or unsubscribing. Without affecting the original business logic, the encapsulated proxy class has the ability to modify the subscription relationship of the first consumer instance.

[0019] In some possible implementations, the agent module can determine the proxy class, determine the first topic subscribed to by the first consumer instance based on the proxy class, and then modify the bytecode related to the subscription relationship in the proxy class so that after the modified proxy class is executed, the first consumer instance does not have a subscription relationship with at least one target topic.

[0020] In this method, through bytecode enhancement technology, the agent module can dynamically modify the subscription relationship of the first consumer instance during runtime, thereby controlling message queue traffic without modifying the original business logic or intruding on the source code.

[0021] Secondly, this application provides a data processing apparatus, which includes an agent module deployed in a first consumer application. The first consumer application and a first message queue are deployed in a first data center. The agent module includes:

[0022] The acquisition unit is configured to acquire the first consumer instance in the first data center in response to a consumer creation operation triggered by the first consumer application; and to acquire the first topic subscribed by the first consumer instance in response to a topic subscription operation triggered by the first consumer application.

[0023] The generation unit is used to encapsulate the first consumer instance to generate a second consumer instance; wherein the second consumer instance includes information indicating the first consumer instance and information indicating the subscription relationship between the first consumer instance and the first topic;

[0024] In response to a failure in the first data center, the processing unit determines, based on the second consumer instance, the first topic subscribed to by the first consumer instance, and unsubscribes from at least one target topic within the first topic, causing the first consumer instance to stop consuming messages associated with at least one target topic in the first message queue cluster.

[0025] In some possible implementations, the second data center deploys a second consumer application and a second message queue cluster. The second message queue cluster belongs to the same message queue cluster as the first message queue cluster. The third consumer instance created by the second consumer application belongs to the same consumer group as the first consumer instance. The processing unit of the agent module is specifically used for:

[0026] Based on the second consumer instance, determine the first topic subscribed to by the first consumer instance, and unsubscribe from all first topics;

[0027] The agent module also includes a rebalancing unit, which is used for:

[0028] Trigger a rebalancing operation for the message queue, so that all messages in the message queue cluster associated with the first topic are consumed by the third consumer instance.

[0029] In some possible implementations, the device also includes a distribution module, which is used for:

[0030] In response to a failure in the first data center, received external requests are distributed to the second data center.

[0031] In some possible implementations, the processing unit of the agent module is specifically used for:

[0032] In response to a failure in the first data center, a set of topics is retrieved, which includes at least one target topic for which subscription needs to be cancelled;

[0033] Based on the second consumer instance, determine the first topic subscribed to by the first consumer instance;

[0034] Query the first topic subscribed to by the first consumer instance based on the topic set, and unsubscribe from at least one target topic in the first topic that matches the topic set.

[0035] In some possible implementations, the acquisition unit of the agent module is also used for:

[0036] Get the subscription status of the first topic;

[0037] The processing unit of the agent module is also used for:

[0038] In response to the existence of unsubscribed topics among the other topics in the first topic, subscribe to the unsubscribed topics; wherein, the other topics include topics in the first topic other than at least one target topic, and the subscription status of the unsubscribed topics is unsubscribed.

[0039] In some possible implementations, the generation unit of the agent module is specifically used for:

[0040] Determine the primitive class of the first consumer instance, which is represented in bytecode form;

[0041] The original class of the first consumer instance is encapsulated to create a proxy class; and the first topic subscribed by the first consumer instance is added to the proxy class.

[0042] Generate a second consumer instance based on the proxy class.

[0043] In some possible implementations, the generation unit of the agent module is specifically used for:

[0044] Determine the proxy class;

[0045] Based on the proxy class, determine the first topic subscribed to by the first consumer instance;

[0046] Modify the bytecode related to the subscription relationship in the proxy class so that when the modified proxy class is executed, the first consumer instance has no subscription relationship with at least one target topic.

[0047] Thirdly, this application provides a computing device including at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory to cause the computing device to perform the data processing method as described in the first aspect or any implementation thereof.

[0048] Fourthly, this application provides a computing device cluster, the computing device cluster including at least one computing device, the at least one computing device including at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory to cause the computing device cluster to perform the data processing method as described in the first aspect or any implementation thereof.

[0049] Fifthly, this application provides a computer-readable storage medium including computer-readable instructions for implementing the data processing method as described in the first aspect or any implementation thereof.

[0050] In a sixth aspect, this application provides a computer program product comprising computer-readable instructions for implementing the data processing method as described in the first aspect or any implementation thereof.

[0051] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0052] To more clearly illustrate the technical method of this application, the accompanying drawings used will be briefly described below.

[0053] Figure 1 A schematic diagram of a local active-active architecture provided for this application;

[0054] Figure 2 A schematic diagram of another active-active architecture within the same city provided in this application;

[0055] Figure 3A schematic diagram of yet another active-active architecture within the same city provided in this application;

[0056] Figure 4 A schematic diagram of the architecture of a data processing system provided in this application;

[0057] Figure 5 This application provides a schematic diagram of the structure of an agent module;

[0058] Figure 6 A schematic diagram of the architecture of another data processing system provided in this application;

[0059] Figure 7 A schematic diagram of the architecture of yet another data processing system provided in this application;

[0060] Figure 8 A flowchart illustrating a data processing method provided in this application;

[0061] Figure 9 A schematic diagram of the structure of a data processing device provided in this application;

[0062] Figure 10 A schematic diagram of the structure of a computing device provided in this application;

[0063] Figure 11 This application provides a schematic diagram of the structure of a computing device cluster;

[0064] Figure 12 This application provides a schematic diagram of another computing device cluster structure.

[0065] Figure 13 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0066] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, the features defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0067] First, let's introduce some of the technical terms used in this application.

[0068] Microservices are a software architecture style that breaks down large, complex business applications into multiple small, independent services. Each service implements a specific business function and supports independent deployment, scaling, and updates, giving business applications advantages such as rapid iteration, high availability, and scalability.

[0069] A data center, also known as a server room or site, refers to a facility used for centralized storage and processing of large amounts of data. Typically, a data center provides servers, storage systems, network equipment, backup systems, cooling systems, and power supplies. Servers can include physical servers and virtual servers; storage systems can include disk arrays, storage area networks (SANs), and network attached storage (NAS); and network equipment can include routers, switches, and firewalls.

[0070] In some examples, microservice-based business applications can be deployed in data centers, which provide the necessary physical and virtual infrastructure to support the deployment and operation of these applications. Typically, each service in a business application can be deployed on a physical or virtual server in the data center, or alternatively, each service in a business application can be deployed in a multi-cloud or hybrid cloud environment.

[0071] In data centers, producer applications, consumer applications, and message queue clusters are typically provided. For example, producer applications, consumer applications, and message queue clusters can be deployed on servers in the data center, or they can be part of a cloud service provided by the data center. Producer applications refer to components or services used to create and send messages to the message queue cluster; consumer applications refer to components or services used to receive and process (also known as consume) messages from the message queue cluster; and the message queue cluster refers to a message queue service using a distributed architecture.

[0072] To illustrate the process in conjunction with the operation of a business application, the producer application creates producer instances, and the consumer application creates consumer instances. Producer instances produce messages for the business application and publish these messages to specific topics in the message queue cluster. Consumer instances subscribe to these topics, receive messages through the message queues of those topics, and execute corresponding business logic, such as data processing, data storage, and data forwarding. The number of messages received by a consumer instance from the message queue cluster can be called message queue traffic. In this way, the processes of producing and consuming messages are decoupled, allowing asynchronous communication between producer and consumer instances. Both producer and consumer instances can independently produce or consume messages.

[0073] During the operation of business applications, there are often scenarios where it is necessary to control the message queue traffic of consumer instances. For example, when a consumer instance subscribes to multiple topics in a message queue cluster, if a data center fails, some topics in the message queue cluster may have difficulty sending and receiving messages normally, requiring control of the message queue traffic of the consumer instance. Another example is in a city-level dual-active architecture; if one data center fails, it is necessary to switch the message queue traffic of consumer instances in the failed data center to the active data center.

[0074] The following explains message queue traffic control in a city-wide active-active architecture. In a city-wide active-active architecture, two or more data centers are deployed in the same geographical location (e.g., the same city). These two or more data centers deploy the same software environment and all provide services to the outside world.

[0075] See Figure 1 The diagram illustrates a dual-active architecture within the same city. Data center A houses producer application A, message queue cluster A, consumer application A, and database A. Data center B houses producer application B, message queue cluster B, consumer application B, and database B. Message queue cluster A and message queue cluster B belong to the same message queue cluster, and consumer instance A and consumer instance B belong to the same consumer group; that is, consumer instance A and consumer instance B subscribe to the same topic.

[0076] During the operation of the business application, producer application A creates producer instance A, producer application B creates producer instance B, consumer application A creates consumer instance A, and consumer application B creates consumer instance B. The access layer distributes external requests generated by the business application to either producer instance A or producer instance B for execution. Producer instance A or producer instance B produces messages related to the business application and uploads them to the message queue cluster. Correspondingly, consumer instance A or consumer instance B pulls messages from the message queue cluster, consumes the messages, and writes the resulting data to the database.

[0077] Because data centers A and B are physically close in a dual-active architecture within the same city, they can use dedicated lines for network connection, resulting in low network latency. Therefore, data generated after consumer instance A or consumer instance B consumes messages can be written to database A in data center A, while database B in data center B is only used for reading data, enabling data synchronization between database A in data center A and database B in data center B.

[0078] In the同城双活架构下, there are scenarios that require traffic switching. For example, when one of the data centers (e.g., Data Center A) fails, the business application can be switched to another data center (e.g., Data Center B) to run, ensuring the high availability and business continuity of the business application. Another example is that during disaster recovery drills, simulating the failure of one of the data centers (e.g., Data Center A), the business application can be switched to another data center (e.g., Data Center B) to run, testing the ability to support all traffic access and the normal operation of the business application under a single-sided architecture.

[0079] See Figure 2 The schematic diagram of another同城双活架构shown. For scenarios that require traffic switching, when Data Center A cannot run and Data Center B can run normally, the access layer can distribute all external requests to the producer instance B of Data Center B and use the database B of Data Center B as the primary database to read and write data.

[0080] However, after all external requests are distributed to Data Center B, the producer instance B produces messages to the message queue cluster. Since the consumer instance A and consumer instance B of Data Center A subscribe to the same topic, the consumer instance A of Data Center A can still consume the messages in the message queue cluster. Therefore, only the control of the access layer traffic can be achieved, and it is difficult to control the message queue traffic in Data Center A.

[0081] Generally, the industry achieves the control of message queue traffic by modifying the business code. Specifically, the developer of the business application can add code logic for stopping and starting the consumption of the consumer instance in the business code as follows:

[0082] private KafkaListenerEndpointRegistry endpointRegistry;

[0083] endpointRegistry.getListenerContainer("<KafkaListener's bean name>").pause();

[0084] endpointRegistry.getListenerContainer("<KafkaListener's bean name>").resume();

[0085] It should be noted that the term "同城双活架构" in the original text may need to be accurately translated according to the specific context, and the above translation is a placeholder for now.Among them, endpointRegistry.getListenerContainer("<Bean name of KafkaListener>").pause() is used to specify that the consumer instance in the data center stops consuming, and endpointRegistry.getListenerContainer("<Bean name of KafkaListener>").resume() is used to specify that the consumer instance in the data center starts consuming.

[0086] See Figure 3 Refer to the schematic diagram of another同城双活架构 (dual-active architecture in the same city) shown. After the business code is modified, the consumer instance A in data center A stops consuming messages, and only the consumer instance B in data center B consumes messages. The access layer traffic and message queue traffic in data center A are cleared. However, the above method requires modifying the business code of the business application, which has a certain degree of invasiveness, a large modification cost, and low efficiency.

[0087] In view of this, the present application provides a data processing method. This method can be executed by a data processing system. Among them, the data processing system includes a first data center, and a first message queue cluster and a first consumer application are deployed in the first data center. A proxy agent module is deployed in the first consumer application.

[0088] Specifically, in response to a consumer creation operation triggered by the first consumer application, the agent module obtains the first consumer instance of the first data center. And, in response to a topic subscription operation triggered by the first consumer application, the agent module obtains the first topic subscribed to by the first consumer instance. Then, the agent module encapsulates the first consumer instance to generate a second consumer instance. Among them, the second consumer instance includes information indicating the first consumer instance and information indicating the subscription relationship between the first consumer instance and the first topic. In response to a failure occurring in the first data center, the agent module determines the first topic subscribed to by the first consumer instance according to the second consumer instance, and cancels the subscription of at least one target topic in the first topic, so that the first consumer instance stops consuming messages associated with at least one target topic in the first message queue cluster.

[0089] It should be noted that the term "同城双活架构" in Chinese is a bit specialized and there may not be a very accurate and common English equivalent. Here, a rough translation is provided for reference. You may need to adjust it according to the specific context and relevant industry jargon.This method deploys an agent module in the consumer application. The agent module intercepts the consumer creation operation, encapsulates the created consumer instance, and stores the topics subscribed to by the consumer instance. Thus, when a data center fails, the agent module can modify the subscription relationship through the encapsulated consumer instance, reducing message consumption by the consumer instance in the failed data center. This achieves message queue flow control without intruding on the original business code, decoupling business logic from message queue flow control logic.

[0090] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0091] See Figure 4 The data processing system architecture diagram shown is shown. The data processing system 40 includes a first data center 41, which is equipped with a first message queue cluster 411 and a first consumer application 412. The first consumer application 412 is equipped with an agent module 413.

[0092] Specifically, the first data center 41 can be understood as a physical location consisting of computing resources, storage systems, network equipment, and other supporting facilities. The first data center 41 can be used to store, process, manage, and distribute data. For example, the first data center 41 can be used for the deployment and operation of business applications based on a microservice architecture.

[0093] The first message queue cluster 411 can be understood as a distributed system for message passing. It provides asynchronous message passing services, loosely coupling the message sender (i.e., producer instance) and the message receiver (i.e., consumer instance), allowing communication between different services (e.g., multiple services in a microservice-based business application system).

[0094] The first consumer application 412 can be understood as a component or service used for receiving and consuming messages (also known as message processing) from the first message queue cluster 411. The first consumer application 412 can be integrated with the first producer application ( Figure 4 (Not shown in the image) The first message queue cluster 411 constitutes the message queue system.

[0095] Agent module 413 can be understood as a program implemented based on bytecode enhancement technology. For example, agent module 413 can be provided as an image, or it can be provided as a cloud service. Bytecode enhancement technology refers to the technique of enhancing the functionality of a business application by modifying bytecode. The execution process of a business application includes compiling source code to obtain bytecode and executing the bytecode to implement business logic. Bytecode enhancement technology obtains bytecode, modifies it, and then executes the modified bytecode to achieve the modification of business logic. Typically, agent module 413 and the business application run in different threads within the same process; that is, agent module 413 is independent of the original business process of the business application, implementing message queue flow control without modifying the source code of the business application.

[0096] The following describes the interaction between the agent module 413 and the first consumer application 412 in this embodiment of the application. See [link to relevant documentation]. Figure 5 The diagram shows the structure of an agent module 413, which includes a bytecode enhancement unit 4131, a consumer buffer 4132, a consumer controller 4133, and a configuration listener 4134. Furthermore, the first data center 41 also deploys a configuration center (…). Figure 4 (Not shown in the image), the configuration center can be understood as a distributed application coordination service, used to provide consistency services for various services in the first data center 41.

[0097] The first consumer application 412 can run the main thread of the business application and the message consumption thread, which belong to the same process. In the message consumption thread, the first consumer application 412 triggers a consumer creation operation. In response to this operation, the agent module (e.g., bytecode enhancement unit 4131) obtains the first consumer instance from the first data center. The first consumer application 412 can also trigger a topic subscription operation. In response, the agent module (e.g., bytecode enhancement unit 4131) obtains the first topic subscribed to by the first consumer instance.

[0098] Next, agent module 412 (e.g., bytecode enhancement unit 4131) encapsulates the first consumer instance, generates a second consumer instance, and stores the second consumer instance in consumer cache 4132 so that agent module 412 can modify the first consumer instance through the second consumer instance.

[0099] When the first data center 41 fails, the configuration center can send a command to the agent module (e.g., configuration listener 4134). In response to the failure of the first data center 41, the agent module determines the first topic subscribed to by the first consumer instance based on the second consumer instance, and unsubscribes from at least one target topic within the first topic. For example, the configuration listener 4134 calls the consumer controller 4133, which retrieves the second consumer instance from the consumer cache 4132, determines the first topic subscribed to by the first consumer instance, and unsubscribes from at least one target topic within the first topic. This causes the first consumer instance to stop consuming messages associated with at least one target topic in the first message queue cluster, thus achieving message queue flow control.

[0100] The data processing system provided in this application embodiment can be applied to a dual-active architecture within the same city. See [link / reference] Figure 6 The diagram illustrates another data processing system architecture. Data processing system 40 includes a first data center 41, which deploys a first message queue cluster 411, a first consumer application 412, a first producer application 414, a first database 415, and a first configuration center 416. The first consumer application 412 includes an agent module 413. A second data center 51 deploys a second message queue cluster 511, a second consumer application 512, a second producer application 514, a second database 515, and a second configuration center 516. The second consumer application 512 also includes an agent module 513. The first message queue cluster 411 and the second message queue cluster 511 belong to the same message queue cluster, and the first configuration center 416 and the second configuration center 516 belong to the same configuration center. Furthermore, in this dual-active architecture within the same city, an access layer for traffic distribution is also included.

[0101] First producer application 414 and second producer application 514 create producer instances. The producer instance created by first producer application 414 produces messages for the business application to first message queue cluster 411, and the producer instance created by second producer application 514 produces messages for the business application to second message queue cluster 511. First consumer application 412 creates a first consumer instance, and second consumer application 512 creates a third consumer instance. The first consumer instance and the third consumer instance belong to the same consumer group, that is, the first consumer instance and the third consumer instance subscribe to the same topic and consume messages using message queues under that topic in the message queue cluster. For example, the first consumer instance and the third consumer instance subscribe to topic A, and topic A in the message queue cluster has message queue 1, message queue 2, message queue 3, and message queue 4. The first consumer instance can consume messages using message queue 1 and message queue 2, and the second consumer instance can consume messages using message queue 3 and message queue 4.

[0102] See Figure 7 The diagram illustrates another data processing system architecture, specifically for scenarios requiring message queue traffic control within a dual-active architecture in the same city, such as disaster recovery drills. Assuming the first data center 41 fails, the access layer distributes all external requests to the unaffected second data center 51. The first configuration center 416 sends a command to the agent module 413 indicating a failure in the first data center 41. The agent module, based on the second consumer instance, determines the first topic subscribed to by the first consumer instance and unsubscribes from all topics within that first topic. Consequently, the first consumer application 412 releases its message queue, clearing the message queue traffic in the first data center 41.

[0103] Furthermore, the agent module 413 can also trigger a rebalancing operation for the message queue, so that all messages associated with the first topic in the message queue cluster are consumed by the third consumer instance, realizing the flow switching of message queue traffic. Without affecting the business continuity of the business application, it ensures that all traffic in the first data center 41 is cleared, thus ensuring the integrity of the disaster recovery exercise.

[0104] based on Figure 4 The data processing system 40 shown in this application also provides a data processing method. The data processing system and method of this application will be described below with reference to embodiments.

[0105] See Figure 8The flowchart shown illustrates a data processing method applied to a data processing system 40. The data processing system 40 includes a first data center 41, on which a first message queue cluster 411 and a first consumer application 412 are deployed. The first consumer application 412 deploys an agent module 413. The method includes the following steps:

[0106] S801: In response to the consumer creation operation triggered by the first consumer application 412, the agent module 413 obtains the first consumer instance of the first data center 41.

[0107] In this embodiment of the application, the first consumer application 412 is equipped with an agent module 413. After the agent module 413 is started and running, it can modify and enhance the relevant functions of the consumer instance created by the first consumer application 412.

[0108] In some embodiments, the agent module 413 can determine the identifier of the first consumer application 412 by obtaining the configuration item information of the first consumer application 412. For example, in the Spring framework, the agent module 413 can determine the name of the first consumer application 412 by obtaining spring.application.name, so that the consumer instance created for the first consumer application 412 can be modified and enhanced subsequently.

[0109] In other embodiments, agent module 413 can determine the region where the first consumer application 412 is located by obtaining the region information of the first consumer application 412, so as to perform batch modification and batch enhancement on different consumer instances created by different consumer applications in that region.

[0110] The consumer creation operation can be understood as the operation used to create consumer instances (also called consumer objects). In some possible implementations, the agent module 413 can obtain the first consumer instance by intercepting aspects. In aspect-oriented programming (AOP), intercepting aspects can be understood as the behavior of inserting extra logic at specific locations in the application's execution (such as method calls or exception handling), and this extra logic is called an aspect.

[0111] In specific implementation, the first consumer application 412 can create a consumer instance. For example, the first consumer application 412 can create a new thread based on the message consumption needs of the business application and trigger a consumer creation operation in the newly created thread. In response to the consumer creation operation triggered by the first consumer application 412, the agent module 413 can obtain the consumer instance (referred to as the first consumer instance in this embodiment) created by the first consumer application 412. For example, when the message queue system including the first message queue cluster and the first consumer application is a Kafka cluster, the first consumer application 412 can call the constructor method new KafkaConsumer() to trigger the consumer creation operation, and the agent module 413 can be configured to intercept the aspect of the constructor method new KafkaConsumer() to obtain the first consumer instance created by the first consumer application 412 in the first data center 41.

[0112] S802: In response to the topic subscription operation triggered by the first consumer application 412, the agent module 413 obtains the first topic subscribed by the first consumer instance.

[0113] The topic subscription operation can be understood as the operation of establishing a subscription relationship between a consumer instance and a topic. Similarly, the agent module 413 can obtain the first topic by intercepting aspects. In specific implementation, the first consumer application 412 can trigger the topic subscription operation. For example, the first consumer application 412 can call the constructor of the subscribed topic to trigger the topic subscription operation. The agent module 413 can be configured to intercept the aspect of the constructor of the subscribed topic to obtain the first topic subscribed by the first consumer instance.

[0114] The embodiments of this application do not limit the number of first topics. Based on the business requirements of the business application, the first topic may include one or more topics.

[0115] S803: Agent module 413 encapsulates the first consumer instance to generate a second consumer instance.

[0116] The encapsulation instance, also known as the wrapper instance, can be understood as the process of encapsulating one instance into another. In the embodiments of this application, the second consumer instance may include information indicating the first consumer instance and information indicating the subscription relationship between the first consumer instance and the first topic.

[0117] In other words, agent module 413 encapsulates the first consumer instance created by first consumer application 412 and the first topic subscribed by the first consumer instance into the second consumer instance, so that the second consumer instance internally holds the first consumer instance and the first topic. In subsequent operation, agent module 413 can modify and enhance the relevant functions of the first consumer instance through the second consumer instance.

[0118] For example, agent module 413 can encapsulate the first consumer instance into a second consumer instance of the ConsumerWrapper class, and store the first topic in the originalTopics property of the second consumer instance as a list.

[0119] S804: In response to a failure in the first data center 41, the agent module 413 determines the first topic subscribed to by the first consumer instance based on the second consumer instance, and unsubscribes from at least one target topic in the first topic, causing the first consumer instance to stop consuming messages associated with at least one target topic in the first message queue cluster 411.

[0120] In this context, a failure in the first data center 41 can be understood as an event occurring in the first data center 41 that requires control over the message queue traffic of the first data center 41. In some embodiments, the first data center 41 may also deploy a configuration center, such as ZooKeeper. The configuration center can establish a connection with the first data center 41 via an Internet Protocol address, and the instruction indicating a failure in the first data center 41 can be sent from the configuration center to the agent module 413.

[0121] Specifically, a node with a first consumer instance can be created in the configuration center. The configuration content of the first consumer instance node is as follows:

[0122] kafkaProhibition:true

[0123] kafkaTopics:

[0124] -kafka-topic-1

[0125] -kafka-topic-2

[0126] Among them, kafkaProhibition is used to indicate whether message queue flow control is required, and kafkaTopics is used to indicate the topics for which message queue flow control is required.

[0127] When the first data center 41 fails, the configuration of the nodes of the first consumer instance in the configuration center can change. For example, the configuration center can monitor the operating status of the first data center 41, and when the operating status indicates that the first data center 41 has failed, it sends a command to the agent module 413. Alternatively, when the first data center 41 fails, the operations and maintenance personnel can modify the configuration of the nodes of the first consumer instance, specifically, change the kafkaProhibition field to true, in order to send a command to the agent module 413.

[0128] Agent module 413 can determine the first topic subscribed to by the first consumer instance by using the information in the encapsulated second consumer instance that indicates the subscription relationship between the first consumer instance and the first topic. Since the second consumer instance includes information indicating the first consumer instance, agent module 413 can also modify the first consumer instance to cancel the subscription relationship between the first consumer instance and at least one target topic. In this way, the first consumer instance stops consuming messages associated with at least one target topic in the first message queue cluster, reducing the traffic of message queues in the first data center 41.

[0129] As described above, in this application, an agent module is mounted in the consumer application. The agent module encapsulates the first consumer instance created by the consumer application, so that when the data center fails, the agent module can dynamically adjust the message consumption behavior of the consumer instance in real time through the encapsulated consumer instance. This achieves control over message queue traffic in the data center without modifying the business source code.

[0130] In some embodiments, agent module 413 can unsubscribe from a subset of target topics. Specifically, in response to a failure in the first data center 41, agent module 413 can obtain a topic set, which includes at least one target topic that needs to be unsubscribed from. Then, agent module 413 can determine the first topic subscribed to by the first consumer instance based on the second consumer instance, query the first topic subscribed to by the first consumer instance based on the topic set, and unsubscribe from at least one target topic in the first topic that matches the topic set.

[0131] In other words, the instruction sent to agent module 413 regarding the failure of the first data center 41 can carry a topic set, which can be used to indicate the target topics that need to be unsubscribed from. For example, agent module 413 can determine the topic set through the kafkaTopics field in the configuration content of the first consumer instance's node in the configuration center. Then, agent module 413 can compare the target topics in the topic set with the first topics subscribed to by the first consumer instance, and unsubscribe from the target topics included in the first topics. For example, if the first consumer instance subscribes to topics 1, 2, and 3, and the target topics in the topic set are topics 1 and 2, then agent module 413 can unsubscribe from topics 1 and 2, while maintaining the subscription relationship between the first consumer instance and topic 3.

[0132] In this way, by providing a set of topics, the agent module can modify the subscription relationship for some topics in the first topic, refine the granularity of message queue traffic control to the topic level, achieve more detailed message queue traffic control, and improve applicability.

[0133] Furthermore, the agent module 413 can also subscribe to topics. Specifically, after the agent module 413 determines the first topic subscribed to by the first consumer instance based on the second consumer instance, and unsubscribes from at least one target topic within the first topic, the agent module 413 can also obtain the subscription status of the first topic. In response to the existence of unsubscribed topics among the other topics in the first topic, the agent module 413 subscribes to the unsubscribed topics. These other topics include topics in the first topic other than at least one target topic, and the subscription status of the unsubscribed topics is "unsubscribed."

[0134] In other words, agent module 413 can not only unsubscribe from the topics of the first consumer instance, but also resubscribe to the topics of the first consumer instance. For example, if the first consumer instance subscribes to topics 1, 2, and 3, and the target topics are topics 1 and 2, and due to a failure of the first data center 41, the subscription relationship between the first consumer instance and topic 3 is abnormally canceled, agent module 413 can obtain the subscription status of topics 1, 2, and 3 after unsubscribing from topics 1 and 2. If the subscription status of topic 3 is unsubscribed, agent module 413 can subscribe to topic 3, thereby restoring the subscription relationship between the first consumer instance and topic 3.

[0135] Thus, after implementing traffic control for the message queue, agent module 413 can also check the subscription relationship of the first consumer instance to ensure that the first consumer instance can normally consume messages using the uncontrolled topics, thus guaranteeing business continuity.

[0136] The data processing method provided in this application embodiment can also be applied to active-active scenarios within the same city. Specifically, the data processing system may further include a second data center, which deploys a second consumer application and a second message queue cluster. The second message queue cluster and the first message queue cluster belong to the same message queue cluster, and the third consumer instance created by the second consumer application belongs to the same consumer group as the first consumer instance.

[0137] In other words, the first consumer instance created by the first consumer application 412 and the third consumer instance created by the second consumer application subscribe to the same topic, and both the first consumer instance and the third consumer instance consume messages through a message queue cluster that includes the first message queue cluster and the second message queue cluster.

[0138] It should be noted that the second data center can also be deployed in an independent data processing system. That is, data processing system A includes the first data center, which deploys the first consumer application and the first message queue cluster, and data processing system B includes the second data center, which deploys the second consumer application and the second message queue cluster.

[0139] In addition, the data processing system may also include an access layer. In response to a failure in the first data center 41, the access layer can distribute received external requests to a second data center. For example, the access layer can distribute received external requests to the second data center using load balancing rules.

[0140] Thus, in a dual-active scenario within the same city, if the first data center 41 fails, or if the first data center 41 fails during a disaster recovery exercise, the access layer can distribute all external requests to the second data center that has not failed, allowing the second data center to perform the message production process and clear the access layer traffic in the first data center.

[0141] In a local active-active scenario, agent module 413 can determine the first topic subscribed to by the first consumer instance based on the second consumer instance, and unsubscribe from all first topics. In other words, in a local active-active scenario, agent module 413 can unsubscribe from all first topics, so that the first consumer instance does not subscribe to topics and does not consume messages in the first message queue cluster 41.

[0142] Furthermore, agent module 413 can also trigger a rebalancing operation for the message queue, so that all messages in the message queue cluster associated with the first topic are consumed by the third consumer instance.

[0143] The message queue rebalancing operation can be understood as the operation of reallocating message queues. In this embodiment, since the agent module 413 unsubscribes from the first topic, the first consumer instance no longer consumes messages. The message queue rebalancing operation triggered by the agent module 413 reallocates the message queue originally used by the first consumer instance to a third consumer instance created by the second consumer application. This allows the third consumer instance to use the message queue originally used by the first consumer instance for message consumption, improving resource utilization and ensuring the business continuity of the application. Furthermore, it achieves the switching of message queue traffic from the first consumer instance to the third consumer instance, ensuring zero message queue traffic in the first data center and ensuring the integrity of disaster recovery exercises.

[0144] In some possible implementations, agent module 413 can implement consumer queue flow control based on bytecode enhancement technology. Taking agent module 413 implemented in Java as an example, during the execution of a Java program, the source code is in .java format. After the compiler compiles the source code, it obtains bytecode in .class format, which can also be called a class file. Then, the Java Virtual Machine (JVM) executes the bytecode, generating class instances (also called class objects). Agent module 413, based on bytecode enhancement technology, can modify the .class bytecode when it is loaded into the JVM, thus allowing the JVM to execute the modified bytecode.

[0145] In this embodiment of the application, agent module 413 can determine the original class of the first consumer instance, wherein the original class is represented in the form of bytecode. agent module 413 encapsulates the original class of the first consumer instance to create a proxy class. agent module 413 adds the first topic subscribed by the first consumer instance to the proxy class. Then, agent module 413 can generate a second consumer instance according to the proxy class.

[0146] In other words, agent module 413 intercepts the consumer creation operation and topic subscription operation of the first consumer application 42, encapsulates the original class of the first consumer instance into a proxy class, and adds the first topic subscribed by the first consumer instance to the proxy class.

[0147] Thus, through instance encapsulation implemented by the agent module based on bytecode enhancement technology, the agent module can add additional functional logic to the proxy class during the encapsulation process, such as adding methods for subscribing or unsubscribing. Without affecting the original business logic, the encapsulated proxy class has the ability to modify the subscription relationship of the first consumer instance.

[0148] Furthermore, during message queue flow control, agent module 413 can determine the proxy class, and based on the proxy class, determine the first topic subscribed to by the first consumer instance. Then, agent module 413 can modify the bytecode related to the subscription relationship in the proxy class so that after the modified proxy class is executed, the first consumer instance does not have a subscription relationship with at least one target topic.

[0149] In other words, agent module 413 modifies the bytecode of the agent class based on bytecode enhancement technology, and when the bytecode is executed, it achieves the purpose of modifying the subscription relationship between the first consumer instance and the first topic.

[0150] Thus, through bytecode enhancement technology, the agent module can dynamically modify the subscription relationship of the first consumer instance during runtime, thereby controlling message queue traffic without modifying the original business logic or intruding on the source code.

[0151] Based on the data processing method of the foregoing embodiments, this application also provides a data processing apparatus. The data processing apparatus will now be described in conjunction with the accompanying drawings.

[0152] See Figure 9 The schematic diagram of the data processing device shown illustrates that the data processing device 90 includes an agent module 91, which is deployed in a first consumer application. The first consumer application and a first message queue are deployed in a first data center. The agent module 91 includes:

[0153] The acquisition unit 911 is configured to acquire the first consumer instance in the first data center in response to a consumer creation operation triggered by the first consumer application; and to acquire the first topic subscribed by the first consumer instance in response to a topic subscription operation triggered by the first consumer application.

[0154] The generation unit 912 is used to encapsulate the first consumer instance to generate a second consumer instance; wherein the second consumer instance includes information indicating the first consumer instance and information indicating the subscription relationship between the first consumer instance and the first topic;

[0155] In response to a failure in the first data center, the processing unit 913 determines, based on the second consumer instance, the first topic subscribed to by the first consumer instance and unsubscribes from at least one target topic in the first topic, causing the first consumer instance to stop consuming messages associated with at least one target topic in the first message queue cluster.

[0156] It should be noted that agent module 91 and agent module 413 mentioned above are the same module, only represented by different labels.

[0157] The aforementioned acquisition unit 911, generation unit 912, and processing unit 913 can be implemented through hardware modules or software modules. Specifically, the acquisition unit 911, generation unit 912, and processing unit 913 can be implemented through a computing device or a computing engine on a computing device. The following explanation uses the generation unit 912 as an example.

[0158] When implemented in software, generation unit 912 can be an application or application module, such as a computing engine, running on a computing device or cluster of computing devices. This application can be provided to users as a virtualization service. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM services can be services that use virtualization technology to create virtual machine (VM) resource pools on multiple physical hosts (such as computing devices) to provide VMs for users to use on demand. BMS services are services that use virtualization technology to create BMS resource pools on multiple physical hosts to provide BMS for users to use on demand. Container services are services that use virtualization technology to create container resource pools on multiple physical hosts to provide containers for users to use on demand. A VM is a simulated virtual computer, that is, a logical computer. A BMS is a scalable, high-performance computing service with computing performance indistinguishable from traditional physical machines and features secure physical isolation. A container is a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service, and container service mentioned above are merely specific examples. In practical applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0159] When implemented in hardware, the generation unit 912 may include at least one computing device, such as a server. Alternatively, the generation unit 912 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0160] In some possible implementations, the second data center deploys a second consumer application and a second message queue cluster. The second message queue cluster belongs to the same message queue cluster as the first message queue cluster. The third consumer instance created by the second consumer application belongs to the same consumer group as the first consumer instance. The processing unit 913 of the agent module 91 is specifically used for:

[0161] Based on the second consumer instance, determine the first topic subscribed to by the first consumer instance, and unsubscribe from all first topics;

[0162] The agent module 91 also includes a rebalancing unit 914, which is used for:

[0163] Trigger a rebalancing operation for the message queue, so that all messages in the message queue cluster associated with the first topic are consumed by the third consumer instance.

[0164] In some possible implementations, the device 90 further includes a distribution module 92, which is used for:

[0165] In response to a failure in the first data center, received external requests are distributed to the second data center.

[0166] In some possible implementations, the processing unit 913 of the agent module 91 is specifically used for:

[0167] In response to a failure in the first data center, a set of topics is retrieved, which includes at least one target topic for which subscription needs to be cancelled;

[0168] Based on the second consumer instance, determine the first topic subscribed to by the first consumer instance;

[0169] Query the first topic subscribed to by the first consumer instance based on the topic set, and unsubscribe from at least one target topic in the first topic that matches the topic set.

[0170] In some possible implementations, the acquisition unit 911 of the agent module 91 is also used for:

[0171] Get the subscription status of the first topic;

[0172] The processing unit 913 of the agent module 91 is also used for:

[0173] In response to the existence of unsubscribed topics among the other topics in the first topic, subscribe to the unsubscribed topics; wherein, the other topics include topics in the first topic other than at least one target topic, and the subscription status of the unsubscribed topics is unsubscribed.

[0174] In some possible implementations, the generation unit 912 of the agent module 91 is specifically used for:

[0175] Determine the primitive class of the first consumer instance, which is represented in bytecode form;

[0176] The original class of the first consumer instance is encapsulated to create a proxy class; and the first topic subscribed by the first consumer instance is added to the proxy class.

[0177] Generate a second consumer instance based on the proxy class.

[0178] In some possible implementations, the generation unit 912 of the agent module 91 is specifically used for:

[0179] Determine the proxy class;

[0180] Based on the proxy class, determine the first topic subscribed to by the first consumer instance;

[0181] Modify the bytecode related to the subscription relationship in the proxy class so that when the modified proxy class is executed, the first consumer instance has no subscription relationship with at least one target topic.

[0182] This application also provides a computing device 1000. For example... Figure 10 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0183] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1002 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0184] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0185] The memory 1006 may include volatile memory, such as random access memory (RAM). The memory 1006 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). The memory 1006 stores executable program code, which the processor 1004 executes to implement the aforementioned data processing method. Specifically, the memory 1006 stores instructions for the data processing device 90 to execute the data processing method.

[0186] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0187] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0188] like Figure 11 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store instructions from the same data processing device 90 for executing data processing methods.

[0189] In some possible implementations, one or more computing devices 1000 in the computing device cluster may also store some instructions of the data processing device 90 for executing data processing methods. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions of the data processing device 90 for executing data processing methods.

[0190] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions for executing some functions of the data processing device 90.

[0191] Figure 12 One possible implementation is shown. For example... Figure 12 As shown, two computing devices 1000A and 1000B are connected via a communication interface 1008. The memory in computing device 1000A stores instructions for executing the functions of agent module 91 (including acquisition unit 911, generation unit 912, processing unit 913, and rebalancing unit 914). The memory in computing device 1000B stores instructions for executing the functions of distribution module 92. In other words, the memory 1006 of computing devices 1000A and 1000B jointly stores instructions from data processing device 90 for executing data processing methods.

[0192] Figure 12 The connection method between the computing device clusters shown can be based on the fact that the data processing device 90 provided in this application includes an agent module 91 and a distribution module 92. Therefore, it is considered that the functions implemented by the acquisition unit 911, the generation unit 912, the processing unit 913 and the rebalancing unit 914 are executed by the computing device 1000A, and the functions implemented by the distribution module 92 are executed by the computing device 1000B.

[0193] It should be understood that Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0194] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 One possible implementation is shown. For example... Figure 13As shown, the two computing devices 1000C and 1000D are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000C stores instructions for performing the functions of agent module 91 (including acquisition unit 911, generation unit 912, processing unit 913, and rebalancing unit 914). Simultaneously, the memory 1006 in computing device 1000D stores instructions for performing the functions of distribution module 92.

[0195] Figure 13 The connection method between the computing device clusters shown can be based on the fact that the data processing apparatus 90 provided in this application includes an agent module 91 and a distribution module 92. Therefore, it is considered that the functions implemented by the acquisition unit 911, the generation unit 912, the processing unit 913, and the rebalancing unit 914 are executed by the computing device 1000C, and the functions implemented by the distribution module 92 are executed by the computing device 1000D. It should be understood that... Figure 13 The functions of the computing device 1000C shown can also be performed by multiple computing devices 1000. Similarly, the functions of the computing device 1000D can also be performed by multiple computing devices 1000.

[0196] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the data processing method described above for a data processing apparatus.

[0197] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the above-described data processing method.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized by, The method is applied to a data processing system, the data processing system comprising a first data center, the first data center being deployed with a first message queue cluster and a first consumer application, an agent module being deployed in the first consumer application, the method comprising: in response to a consumer creation operation triggered by the first consumer application, the agent module obtaining a first consumer instance of the first data center; and in response to a topic subscription operation triggered by the first consumer application, the agent module obtaining a first topic to which the first consumer instance subscribes; the agent module encapsulating the first consumer instance to generate a second consumer instance; wherein the second consumer instance comprises information indicating the first consumer instance and information indicating a subscription relationship between the first consumer instance and the first topic; in response to a failure of the first data center, the agent module determining the first topic to which the first consumer instance subscribes according to the second consumer instance, and unsubscribing at least one target topic in the first topic, so that the first consumer instance stops consuming messages in the first message queue cluster associated with the at least one target topic.

2. The method of claim 1, wherein, The data processing system further comprises a second data center, the second data center being deployed with a second consumer application and a second message queue cluster, the second message queue cluster belonging to the same message queue cluster as the first message queue cluster, a third consumer instance created by the second consumer application belonging to the same consumer group as the first consumer instance; the agent module determining the first topic to which the first consumer instance subscribes according to the second consumer instance, and unsubscribing at least one target topic in the first topic, comprising: the agent module determining the first topic to which the first consumer instance subscribes according to the second consumer instance, and unsubscribing all the first topics; The method further comprises: the agent module triggering a rebalancing operation for a message queue, so that messages in the message queue cluster associated with the first topic are all consumed by the third consumer instance.

3. The method of claim 2, wherein, The data processing system further comprises an access layer, and in response to the failure of the first data center, the method further comprises: the access layer distributing a received external request to the second data center.

4. The method of claim 1, wherein, In response to the failure of the first data center, the agent module determines the first topic to which the first consumer instance subscribes according to the second consumer instance, and unsubscribes at least one target topic in the first topic, comprising: in response to the failure of the first data center, the agent module obtaining a topic set, the topic set comprising at least one target topic that needs to be unsubscribed; the agent module determines the first topic to which the first consumer instance subscribes according to the second consumer instance; The agent module subscribes to the first topic subscribed by the first consumer instance according to the topic set, and unsubscribes from the at least one target topic in the first topic that hits the topic set.

5. The method according to claim 1 or 4, characterized in that, After the agent module determines the first topic subscribed by the first consumer instance according to the second consumer instance and unsubscribes from the at least one target topic in the first topic, the method further comprises: The agent module obtains the subscription state of the first topic; In response to the existence of an unsubscribed topic in other topics in the first topic, the agent module subscribes to the unsubscribed topic; wherein the other topics include topics in the first topic other than the at least one target topic, and the subscription state of the unsubscribed topic is unsubscribed.

6. The method according to any one of claims 1 to 5, characterized in that, The agent module encapsulates the first consumer instance to generate a second consumer instance, comprising: The agent module determines the original class of the first consumer instance, which is represented in the form of bytecode; The agent module encapsulates the original class of the first consumer instance to create a proxy class; and the agent module adds the first topic subscribed by the first consumer instance to the proxy class; The agent module generates a second consumer instance according to the proxy class.

7. The method of claim 6, wherein, The agent module determines the first topic subscribed by the first consumer instance according to the second consumer instance and unsubscribes from the at least one target topic in the first topic, comprising: The agent module determines the proxy class; The agent module determines the first topic subscribed by the first consumer instance according to the proxy class; The agent module modifies the bytecode related to the subscription relationship in the proxy class, so that after the modified proxy class is executed, the first consumer instance does not have a subscription relationship with the at least one target topic.

8. A data processing apparatus, characterized by, The device comprises an agent module, the agent module is deployed in a first consumer application, the first consumer application and a first message queue are deployed in a first data center, and the agent module comprises: The acquisition unit is configured to, in response to a consumer creation operation triggered by the first consumer application, acquire a first consumer instance of the first data center; and in response to a topic subscription operation triggered by the first consumer application, acquire a first topic subscribed by the first consumer instance; The generation unit is configured to encapsulate the first consumer instance to generate a second consumer instance; wherein the second consumer instance comprises information indicating the first consumer instance and information indicating a subscription relationship between the first consumer instance and the first topic; The processing unit is configured to, in response to the failure of the first data center, determine, according to the second consumer instance, first topics to which the first consumer instance subscribes, and unsubscribe from at least one target topic in the first topics, so that the first consumer instance stops consuming messages associated with the at least one target topic in the first message queue cluster.

9. The apparatus of claim 8, wherein, The second data center is deployed with a second consumer application and a second message queue cluster, the second message queue cluster belongs to the same message queue cluster as the first message queue cluster, and a third consumer instance created by the second consumer application belongs to the same consumer group as the first consumer instance; the processing unit of the agent module is specifically configured to: determine, according to the second consumer instance, first topics to which the first consumer instance subscribes, and unsubscribe from all the first topics; The agent module further includes a rebalancing unit, and the rebalancing unit of the agent module is configured to: trigger a rebalancing operation for a message queue, so that messages associated with the first topics in the message queue cluster are all consumed by the third consumer instance.

10. The apparatus of claim 9, wherein, The apparatus further includes a distribution module, and the distribution module is configured to: in response to the failure of the first data center, distribute a received external request to the second data center.

11. The apparatus of claim 8, wherein, The processing unit of the agent module is specifically configured to: in response to the failure of the first data center, obtain a topic set, the topic set including at least one target topic that needs to be unsubscribed; determine, according to the second consumer instance, first topics to which the first consumer instance subscribes; query the first topics to which the first consumer instance subscribes according to the topic set, and unsubscribe from the at least one target topic in the first topics that hits the topic set.

12. The apparatus of claim 8 or 11, wherein, The obtaining unit of the agent module is further configured to: obtain a subscription state of the first topics; The processing unit of the agent module is further configured to: in response to the existence of an unsubscribed topic in other topics in the first topics, subscribe to the unsubscribed topic; wherein the other topics include topics in the first topics other than the at least one target topic, and the subscription state of the unsubscribed topic is unsubscribed.

13. The apparatus of any one of claims 8 to 12, wherein, The generating unit of the agent module is specifically configured to: determine an original class of the first consumer instance, the original class being represented in the form of bytecode; encapsulate the original class of the first consumer instance to create a proxy class, and add the first topics to which the first consumer instance subscribes to the proxy class; generate the second consumer instance according to the proxy class.

14. The apparatus of claim 13, wherein, The generating unit of the agent module is specifically configured to: determine the proxy class; determine, according to the proxy class, first topics to which the first consumer instance subscribes; modify bytecode related to a subscription relationship in the proxy class, so that the first consumer instance and the at least one target topic do not have a subscription relationship after the modified proxy class is executed.

15. A computing device, comprising: The computing device comprises at least one processor and at least one memory having computer-readable instructions stored therein; the at least one processor executes the computer-readable instructions to cause the computing device to perform the method of any one of claims 1 to 7.

16. A cluster of computing devices, characterized in that, The computing device cluster comprises at least one computing device, the at least one computing device comprising at least one processor and at least one memory having computer-readable instructions stored therein; the at least one processor executes the computer-readable instructions to cause the computing device cluster to perform the method of any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises computer-readable instructions for implementing the method of any one of claims 1 to 7.

18. A computer program product, characterised in that, The computer program product comprises computer-readable instructions for implementing the method of any one of claims 1 to 7.

Images