Data processing method, electronic device, storage medium and program product

Abstract

The present disclosure provides a data processing method, an electronic device, a storage medium and a program product, the method comprising: obtaining a message reading configuration parameter, the message reading configuration parameter including type indication information; determining a service interface according to the type indication information; and processing a message listening component of a message queue by calling the service interface. The scheme provided by the embodiment of the present disclosure utilizes the message reading configuration parameter to determine the service interface, and processes the message listening component by calling the service interface, thereby eliminating the need to manually modify the code, and since the message listening component is processed, there is no need to restart the entire message queue, and there is no need to restart the message queue.

Description

Background Art

[0002] When the message queue is in the process of service operation, for example, when the processing capacity of the message queue is insufficient due to a business peak, the processing capacity of the message queue needs to be adjusted. The related technology is to adjust the processing capacity of the message queue by manually modifying the code. After manually modifying the code, it is necessary to restart the message queue to implement the adjustment of the processing capacity of the message queue. In an online environment, restarting the message queue has a greater impact.

[0003] Therefore, how to adjust the message queue without restarting the message queue has become one of the hot issues in current research.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute relevant technology known to ordinary technicians in the field. Summary of the invention

[0005] The embodiments of the present disclosure provide a data processing method, an electronic device, a storage medium, and a program product, which can adjust a message queue without manually modifying the code and restarting the message queue.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a data processing method is provided, including: obtaining message reading configuration parameters, the message reading configuration parameters including type indication information; determining a service interface according to the type indication information; and processing a message listening component of a message queue by calling the service interface.

[0008] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data processing methods by executing the executable instructions.

[0009] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method described in any one of the above is implemented.

[0010] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, any one of the above-mentioned data processing methods is implemented.

[0011] The data processing method, electronic device, storage medium and program product provided in the embodiments of the present disclosure determine the service interface by obtaining the type indication information in the message reading configuration parameters, and process the message listening component of the message queue by calling the service interface. On the one hand, there is no need to manually modify the code. On the other hand, the message listening component is the execution unit of the task in the message queue. The processing capacity of the message queue is determined by the capacity of the message listening component. If the processing capacity of the message queue needs to be adjusted due to a business peak, the message listening component is processed by calling the service interface, thereby realizing the adjustment of the processing capacity of the message queue. Moreover, since the message listening component is processed, there is no need to restart the entire message queue, and the processing capacity can be dynamically adjusted and configured in real time, so as to timely improve the consumption capacity of the message queue and avoid message accumulation.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A schematic diagram showing the system structure of a data processing method in an embodiment of the present disclosure.

[0015] Figure 2 A schematic diagram of a data processing method in an embodiment of the present disclosure is shown.

[0016] Figure 3 A schematic diagram of a front-end visualization process of a data processing method in an embodiment of the present disclosure is shown.

[0017] Figure 4 A data processing method architecture diagram in an embodiment of the present disclosure is shown.

[0018] Figure 5 A schematic diagram of a data processing device in an embodiment of the present disclosure is shown.

[0019] Figure 6 A structural block diagram of an electronic device showing a data processing method in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0021] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0022] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0023] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the data processing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0024] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0025] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is typically the Internet, but may also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) may be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may also be used to replace or supplement the above data communication technologies.

[0026] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0027] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0028] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0029] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0030] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0031] Under the above system architecture, a data processing method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0032] In some embodiments, the data processing method provided in the embodiments of the present disclosure may be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the data processing method provided in the embodiments of the present disclosure may be executed by a server in the above-mentioned system architecture; in other embodiments, the data processing method provided in the embodiments of the present disclosure may be implemented by a terminal device and a server in the above-mentioned system architecture through interaction.

[0033] Message Queue (MQ) is one of the important components in distributed systems, used to transmit information between different application components. It enables asynchronous communication between services, that is, after the sender sends a message to the queue, it can continue to perform other tasks without waiting for the receiver's response. This mechanism improves the decoupling and scalability of the system.

[0034] In a distributed messaging system, it includes: a message producer (Producer), a message queue and a message consumer (Consumer).

[0035] A message producer is an entity responsible for creating and sending messages to a message queue. The producer encapsulates the message into a specific format and sends it to a specified message queue. A producer can send messages to multiple different queues or select a specific queue based on business logic.

[0036] Message queues are used as middleware between message producers and message consumers to store messages from producers. Message queues are a temporary storage place for messages until they are processed by message consumers. Queues can ensure the order, persistence, and reliability of messages. Depending on the implementation, some queues also support complex message routing rules, priority settings, and other features. Message queues also act as a buffer between producers and consumers, so that both do not need to work online at the same time, thus achieving asynchronous communication.

[0037] A message consumer is an entity that reads messages from a message queue and processes them. A message consumer is usually part of an application and is responsible for performing tasks corresponding to the received messages. A message queue can have multiple consumers subscribed to improve processing efficiency; in this case, load balancing is usually used to distribute messages to different consumer instances. After the message consumer completes message processing, it will confirm that the message has been successfully processed, at which point the message can be removed from the message queue. If processing fails, you can decide whether to re-queue or enter the dead letter queue based on the configuration.

[0038] When messaging in a distributed messaging system, the interaction between message producers, message queues, and message consumers includes:

[0039] The message producer sends the message to the message queue;

[0040] The message queue runs the service, receives messages and stores them, waiting for message consumers to retrieve them;

[0041] The message consumer pulls the message from the message queue (or receives the message by push), and then starts processing;

[0042] After processing is completed, the message consumer sends a confirmation signal to the queue to inform that the message has been processed;

[0043] If necessary, message consumers can also reject messages, which may cause the message to be requeued or transferred elsewhere.

[0044] When the message queue is in the service running state, when the message queue receives the message produced by the message producer, the message queue is the consumer of the message producer, that is, in the binary relationship between the message producer and the message queue, the message producer generates the message, the message queue receives and consumes the message, and the message queue is the consumer of the message producer. The form of consuming the message by the message queue is to cache the message. The cached message is used to wait for subsequent message consumers to read or send the cached message to the message consumer according to the set rules.

[0045] In the related art, before the message queue transmits the message, the corresponding pre-configured processing capacity is determined according to the historical data; according to the corresponding pre-configured processing capacity, the code is manually modified to configure the parameters of the message queue, and the pre-configured processing capacity of the message queue is implemented by restarting the message queue. The message queue after restarting is in the service running state and receives the message sent by the message producer. If a sudden business peak is encountered during the message transmission process, and the processing capacity of the message queue is insufficient to cope with the business peak, then the processing capacity of the message queue needs to be adjusted to adapt to the business peak. The related art is to modify the message queue parameters by manually modifying the code, and then to adjust the processing capacity of the message queue. However, manually modifying the code also requires restarting the entire message queue to implement the processing capacity adjustment. In an online environment, restarting the entire message queue will stop the entire message transmission, and the sudden business peak will cause the accumulation of message blockage and even the loss of messages, which is a more serious accident for the online environment. Generally, it is not allowed to restart the message queue in the online environment. Therefore, the technical solution of the embodiment of the present disclosure is to adjust the message consumption processing capacity without restarting the message queue in the online environment.

[0046] Figure 2 A schematic diagram of a data processing method in an embodiment of the present disclosure is shown. Figure 2 As shown, the data processing method provided in the embodiment of the present disclosure includes the following steps.

[0047] Step S202: Acquire message reading configuration parameters, where the message reading configuration parameters include type indication information.

[0048] In the embodiments of the present disclosure, message reading configuration parameters refer to parameters configured to indicate how to consume (i.e., read) messages in a message queue, such as indicating whether to increase the ability to consume messages in a message queue, or to decrease the ability to consume messages in a message queue.

[0049] In some embodiments, the message reading configuration parameters can be obtained from a front-end visualization page. The front-end visualization page can be displayed on a display screen of a terminal device. The front-end visualization page includes a variety of controls. In response to the operation of some or all of the multiple controls, the message reading configuration parameters are configured, and then the message reading configuration parameters are sent to the server. The server receives the message reading configuration parameters.

[0050] In the embodiment of the present disclosure, multiple (two or more) types of indication information are configured in the front-end visualization page, wherein different types of indication information are used to indicate different processing of the message monitoring component.

[0051] In an exemplary embodiment, the type indication information indicating the processing of the message listening component includes: indicating to start or stop the message listening component, indicating to adjust the parameters of the message listening component, and indicating to create the message listening component in the message queue. When dealing with business peaks, starting or stopping the message listening component, adjusting the parameters of the message listening component, and / or creating the message listening component in the message queue can all improve the processing capacity of the message queue, thereby dealing with business peaks.

[0052] Step S204: Determine the service interface according to the type indication information.

[0053] In the embodiments of the present disclosure, different types of indication information correspond to different service interfaces. Different service interfaces can implement different processing of the message listening component of the message queue. That is, the service interface in the embodiments of the present disclosure refers to a functional component that processes the message listening component in the message queue, such as starting or stopping, adjusting parameters, and creating a message listening component.

[0054] Step S206: Process the message monitoring component of the message queue by calling the service interface.

[0055] The disclosed embodiment determines the service interface by obtaining the type indication information in the message reading configuration parameters, and processes the message listening component of the message queue by calling the service interface. On the one hand, there is no need to manually modify the code. On the other hand, the message listening component is the execution unit of the task in the message queue. The processing capacity of the message queue is determined by the capacity of the message listening component. If the processing capacity of the message queue needs to be adjusted due to a business peak, the message listening component is processed by calling the service interface, thereby realizing the adjustment of the processing capacity of the message queue. Moreover, since the message listening component is processed, there is no need to restart the entire message queue, which realizes the dynamic adjustment and real-time configuration of the processing capacity of the message queue without restarting, and timely improves the consumption capacity of the message queue to avoid message accumulation.

[0056] Figure 3 A schematic diagram of a front-end visualization page of an embodiment of the present disclosure is shown. Figure 3 As shown, the front-end visualization page 31 includes at least three configuration scenario options: message listening component start and stop management 311, message listening component parameter adjustment 312, and message listening component addition 313. Among them, message listening component start and stop management 311 is used to start or stop the message listening component according to business needs; message listening component parameter adjustment 312 is used to adjust the parameters of the message listening component according to business needs; message listening component addition 313 is used to create a message listening component operation in the message queue according to business needs. Among them, business needs are determined according to different scenarios.

[0057] When the message listening component start and stop management 311 is clicked and triggered, a message reading configuration parameter is generated in the front-end visualization page 31 , and the message reading configuration parameter includes type indication information indicating whether to start or stop the message listening component.

[0058] When the message listening component parameter adjustment 312 is clicked and triggered, a message reading configuration parameter is generated in the front-end visualization page 31 , and the message reading configuration parameter includes type indication information indicating the adjustment of the parameters of the message listening component.

[0059] When the message listening component addition 313 is clicked and triggered, a message reading configuration parameter is generated in the front-end visualization page 31 , and the message reading configuration parameter includes type indication information indicating the creation of a message listening component in the message queue.

[0060] Among them, there is a backend corresponding to the front-end visualization page; there is a data transmission channel between the front-end and the back-end, for example, a transmission channel established by HTTP (HyperText Transfer Protocol). The message reading configuration parameters of the front-end visualization page are passed to the back-end via HTTP. The back-end runs the data processing method of this embodiment. The back-end receives the message reading configuration parameters. A tuning processor is provided in the back-end, and a thread is opened separately by the tuning processor to asynchronously process the acquired message reading configuration parameters. In the process of transmitting the type indication information from the front end to the back end, in order to ensure the efficiency of data transmission, the type field can be used. In one case, the type field includes: a first type field, a second type field and a third type field. Among them, the first type field is used to indicate the start or stop of the message listening component, the second type field is used to indicate the adjustment of the parameters of the message listening component, and the third type field is used to indicate the creation of the message listening component in the message queue.

[0061] In the embodiment, the above three message listening component start and stop management, message listening component parameter adjustment and message listening component newly added configuration scenario options can also be made into three different configurations of the configuration center. The configuration center can be Nacos (Naming and Configuration Service, an open source dynamic service discovery, configuration management and service management platform) or Apollo (a distributed configuration center), which monitors changes in the above three different configurations. When a configuration change is detected, the specific changed configuration is located to determine the corresponding configuration scenario option.

[0062] By visualizing options for different configuration scenarios or making different configurations on the front end, the problems of slow efficiency and high error rate caused by manual code modification are avoided. The message listening component can be directly processed in accordance with different configuration scenarios without receiving modification codes, which improves the efficiency of message listening component configuration and ensures the correct execution and effectiveness of message queue capacity adjustment.

[0063] In an embodiment, based on the above-mentioned multiple type indication information, there are also multiple corresponding service interfaces. When the type indication information indicates to start or stop the message listening component, the corresponding service interface is a start-stop service interface. When the type indication information indicates to adjust the parameters of the message listening component, the corresponding service interface is a parameter adjustment service interface. When the type indication information indicates to create a message listening component in a message queue, the corresponding service interface is a new service interface.

[0064] In the embodiment, the message monitoring component of the message queue is processed by calling the service interface, including: executing the message monitoring component, or adjusting the message monitoring component, or creating the message monitoring component in the message queue by calling the service interface.

[0065] Based on the above-mentioned multiple different service interfaces, if the service interface is a start-stop service interface, the message listening component is executed by calling the start-stop service interface. If the service interface is a parameter adjustment service interface, the message listening component is adjusted by calling the parameter adjustment service interface. If the service interface is a new service interface, a message listening component is created in the message queue by calling the new service interface. Among them, executing the message listening component is to execute the start component or stop component on the message listening component. Adjusting the message listening component is to adjust the parameters of the message listening component. Creating a message listening component in the message queue is to create corresponding message listening components in the message queue according to the number of message listening components to be created.

[0066] There are many types of message queues in practical applications, such as Kafka, RocketMQ, Apache Pulsar, RabbitMQ, and Pravega. This embodiment takes Kafka as an example of a practical application of a message queue to illustrate the specific process of the data processing method. Kafka service is a distributed message system located between producers and consumers. Producers produce various messages, and consumers consume, process, and analyze these messages. Kafka service is a message system located between producers and consumers for communication, realizing seamless connection between producers and consumers.

[0067] When Kafka is used as a message queue, when the processing capacity of Kafka needs to be adjusted during business peaks, the relevant technology is to adjust the processing capacity by manually modifying the code and restarting Kafka. Furthermore, if the start and stop of Kafka needs to be managed, it is also necessary to manually modify the code or restart Kafka through switch configuration to achieve this, but Kafka generally does not allow restart in an online environment.

[0068] In Kafka, the functional unit corresponding to the message listening component is the consumer listening container instance (KafkaMessageListenerContainer). As the functional execution unit for message delivery and message consumption in Kafka, when the processing capacity of Kafka needs to be adjusted, the consumer listening container instance needs to be processed accordingly.

[0069] In Kafka, the configuration scenarios are divided into Kafka start and stop management, Kafka parameter tuning and Kafka container expansion; among them, Kafka start and stop management corresponds to message listening component start and stop management, and the corresponding type indication information is to indicate to start or stop the consumer listening container instance, Kafka parameter tuning corresponds to message listening component parameter adjustment, and the corresponding type indication information is to indicate to adjust the parameters of the consumer listening container instance, and Kafka container expansion corresponds to message listening component addition, and the corresponding type indication information is to indicate to create a consumer listening container instance in Kafka. In Kafka, the service interface includes Kafka start and stop service interface, Kafka parameter adjustment service interface and Kafka new service interface. Among them, Kafka start and stop service interface corresponds to start and stop service interface, Kafka parameter adjustment service interface corresponds to parameter adjustment service interface, and Kafka new service interface corresponds to new service interface. Processing the consumer listening container instance includes executing the consumer listening container instance, adjusting the consumer listening container instance or creating the consumer listening container instance in Kafka. Among them, executing the consumer listening container instance corresponds to executing the message listening component, adjusting the consumer listening container instance corresponds to adjusting the message listening component, and creating the consumer listening container instance in Kafka corresponds to creating the message listening component in the message queue.

[0070] In the embodiment, a Kafka tuning processor kafkaHandler (Param param) is configured in Kafka, where Param and param represent parameter classes of the Kafka tuning processor. The Kafka tuning processor is a type of tuning processor. A separate thread is opened in the Kafka service through the Kafka tuning processor. The Kafka tuning processor and the consumer listening container instance that normally consumes messages belong to different threads, so the Kafka tuning processor can asynchronously process message reading configuration parameters.

[0071] In order to enable the Kafka tuning process to have the ability to process message reading configuration parameters, an abstract class of Kafka service (AbstractKafkaHandlerService) is defined in the Kafka tuning processor. The AbstractKafkaHandlerService belongs to a public service abstract class in the JAVA language. The Kafka tuning method dealKafka() is provided in the abstract class of Kafka service. After the abstract class of Kafka service is defined, the subclass of the abstract class of Kafka service is determined; the abstract class of Kafka service includes multiple subclasses, and the abstract class of Kafka service and the subclass of the abstract class of Kafka service are in a parent-child relationship, the abstract class of Kafka service is the parent, the subclass of the abstract class of Kafka service is the child, and the subclass of the abstract class of Kafka service inherits the abstract class of Kafka service.

[0072] In the embodiment, the backend also provides a proxy service class, which is configured after the Kafka tuning processor.

[0073] The proxy service class selects different types of indication information through the strategy mode to realize the processing of different consumer listening container instances. Specifically, the proxy service class realizes the abstract class of Kafka service according to the message reading configuration parameters. Since the abstract class of Kafka service belongs to the public service abstract class, and the Kafka tuning method provided by the abstract class of Kafka service belongs to the public method, the Kafka tuning method is empty. When the abstract class of Kafka service is implemented to rewrite the Kafka tuning method, the processing logic of the subclass of the abstract class of Kafka service is written into the abstract class of Kafka service it inherits, and the processing logic of the subclass of the abstract class of Kafka service is realized through different service interfaces, that is, the processing logic in different service interfaces is written into the abstract class of Kafka service, and the consumer listening container instance of Kafka is processed by calling the service interface.

[0074] After the Kafka service obtains the message from the producer, it performs the message consumption operation through the consumer listening container instance. There are multiple consumer listening container instances in the Kafka service. When it is necessary to start and stop the Kafka service, start or stop a certain consumer listening container instance or all consumer listening container instances, so there is no need to restart the entire Kafka service. When the consumption capacity is insufficient due to the business peak, one way is to first adjust the parameters of the existing consumer listening container instances in the Kafka service to improve the consumption capacity of the Kafka service; for example, by adjusting the parameters of the existing consumer listening container instances in the Kafka service, the parameters of the existing consumer listening container instances are more optimized to improve the message processing capacity and adapt to the business peak. Furthermore, the Kafka consumer container can be expanded to increase the consumers of the message to improve the overall consumption capacity of the Kafka service; for example, by adding a new consumer listening container instance on the basis of the existing consumer listening container instance in the Kafka service to achieve the expansion of the Kafka consumer container. By improving the message processing capabilities of existing consumer listening container instances in the Kafka service and expanding the capacity to add new consumer listening container instances, we can effectively cope with business peaks, promptly improve the consumption capacity of the Kafka service, and avoid message backlogs.

[0075] In an embodiment, the process of message consumption by the Kafka service includes: starting the Kafka service, scanning the listener annotation and creating a message listening container instance, wherein the listener annotation is modified on the Kafka service; using the message listening container instance to create a consumer listening container instance; and starting the consumer listening container instance to execute message consumption.

[0076] The Kafka service uses the listener annotation (@KafkaListener) to perform message consumption. The listener annotation is modified on the Kafka service. The listener annotation is based on the spring-kafka framework. After starting the Kafka service, the listener annotation will be scanned and a message listener container instance (ConcurrentMessageListenerContainer) will be created. The message listener container instance is used to create one or more consumer listener container instances (KafkaMessageListenerContainer) through delegation. The created consumer listener container instance provides multi-threading capabilities and implements the functions of multiple consumers. Then the consumer listener container instance is started to perform message consumption and consume the messages generated by the producer. Spring-kafka generates a consumer listener container instance and completes the message consumption by scanning the annotation. Among them, spring-kafka is a real-time data stream processing application based on Kafka. The Kafka service performs message consumption through the consumer listener container instance. If the processing capacity of the Kafka service is adjusted, it is actually a separate adjustment for the consumer listener container instance, so there is no need to restart the entire Kafka service.

[0077] In the embodiment, the message monitoring component includes a start component and a stop component; the service interface includes a start-stop service interface; the start component is used to change the message monitoring component from a stop state to a start-up running state; and the stop component is used to change the message monitoring component from a running state to a stop state.

[0078] Wherein, determining the service interface according to the type indication information includes:

[0079] If the type indication information indicates starting or stopping the message listening component, determining that the service interface is a start-stop service interface;

[0080] Among them, by calling the service interface, the message monitoring component is executed, including:

[0081] Get the message listening component of the message queue by calling the start-stop service interface, and execute the start component or stop component in the message listening component.

[0082] One way to adjust the processing capacity of the message queue is to start or stop the message listening component. Starting the message listening component will increase the processing capacity of the message queue to cope with business peaks, while stopping the message listening component will reduce the processing capacity of the message queue, thereby reducing energy consumption.

[0083] In the message queue, each message listening component includes a start component and a stop component. If the type indication information indicates starting or stopping the message listening component, then the service interface corresponding to the type indication information is the start-stop service interface. If all message listening components in the message queue are to be started or stopped, then all message listening components in the message queue are obtained by calling the start-stop service interface, and the start component or stop component in the message listening component is executed to start or stop the message listening component.

[0084] Taking Kafka as a message queue as an example, each consumer listening container instance includes a start method and a stop method. The start method belongs to the start component, and the stop method belongs to the stop component. If the type indication information indicates to start or stop the consumer listening container instance, then the service interface corresponding to the type indication information is the start-stop service interface. The consumer listening container instance in Kafka is obtained by calling the start-stop service interface, and the start method or stop method of the consumer listening container instance is executed; wherein, when the type indication information indicates to start the consumer listening container instance, the start method of all consumer listening container instances in Kafka is executed, and all consumer listening container instances in Kafka are started. When the type indication information indicates to stop the consumer listening container instance, the stop method of all consumer listening container instances in Kafka is executed, and all consumer listening container instances in Kafka are stopped.

[0085] Specifically, since all consumer listening container instances are registered in the message listening site (KafkaListenerEndpointRegistry), all consumer listening container instances can be obtained from the message listening site.

[0086] Based on the aforementioned abstract class of Kafka service and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When executing the start method or stop method of the consumer listening container instance, the start method or stop method of the consumer listening container instance is written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service, so as to implement the abstract class of Kafka service and rewrite the Kafka tuning method, thereby starting or stopping all consumer listening container instances in Kafka.

[0087] This embodiment can start or stop all message monitoring components in the message queue according to the type indication information without restarting the message queue.

[0088] In the embodiment, the message monitoring component includes a start component and a stop component; the service interface includes a start-stop service interface;

[0089] The message reading configuration parameters also include the object name corresponding to the message listening component;

[0090] Wherein, determining the service interface according to the type indication information includes:

[0091] If the type indication information indicates starting or stopping the message listening component, determining that the service interface is a start-stop service interface;

[0092] Among them, by calling the service interface, the message monitoring component is executed, including:

[0093] By calling the start-stop service interface and the object name corresponding to the message listening component, the message listening component to be started or stopped is determined, and the start component in the message listening component to be started is executed, or the stop component in the message listening component to be stopped is executed.

[0094] If you want to start or stop one or more message listening components in the message queue, you need to locate the corresponding message listening component. Each message listening component has an object name (also known as a Bean name). The object name can be used to quickly and accurately locate the message listening component.

[0095] In the message queue, each message listening component includes a start component and a stop component. If the type indication information indicates starting or stopping one or some message listening components, the service interface corresponding to the type indication information is a start-stop service interface.

[0096] Furthermore, the object name corresponding to the message listening component is obtained from the message reading configuration parameter, and the message listening component to be started or the message listening component to be stopped can be confirmed from the message queue by calling the start-stop service interface and the object name corresponding to the message listening component. The start component in the message listening component to be started is executed to start the message listening component to be started; or the stop component in the message listening component to be stopped is executed to stop the message listening component to be stopped.

[0097] Taking Kafka as a message queue, each consumer listening container instance includes a start method and a stop method. The start method belongs to the start component, and the stop method belongs to the stop component. If the type indication message indicates to start or stop one or some consumer listening container instances, the service interface corresponding to the type indication message is the start-stop service interface.

[0098] Get the object name corresponding to the consumer listening container instance from the message reading configuration parameters. The object name can be used to locate the consumer listening container instance to be started or to be stopped in Kafka.

[0099] By calling the start-stop service interface and the object name corresponding to the consumer listening container instance, the consumer listening container instance to be started or the consumer listening container instance to be stopped is determined from Kafka, and the start method of the consumer listening container instance to be started is executed to start the consumer listening container instance to be started in Kafka; or, the stop method of the consumer listening container instance to be stopped is executed to stop the consumer listening container instance to be stopped in Kafka.

[0100] Based on the aforementioned abstract class of Kafka service and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When executing the start method or stop method of the consumer listening container instance, the start method or stop method of the consumer listening container instance is written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service, and the object name corresponding to the consumer listening container instance is also written into the Kafka tuning method, so as to implement the abstract class of Kafka service and rewrite the Kafka tuning method, so as to start the consumer listening container instance to be started in Kafka or stop the consumer listening container instance to be stopped.

[0101] This embodiment can determine the message listening component to be started or the message listening component to be stopped from the message queue based on the object name corresponding to the message listening component, so as to start the message listening component to be started or stop the message listening component to be stopped without restarting the message queue.

[0102] In the embodiment, the message monitoring component includes a start component and a stop component; the service interface includes a start-stop service interface;

[0103] The message reading configuration parameters also include the message topic name;

[0104] Wherein, determining the service interface according to the type indication information includes:

[0105] If the type indication information indicates starting or stopping the message listening component, determining that the service interface is a start-stop service interface;

[0106] Among them, by calling the service interface, the message monitoring component is executed, including:

[0107] By calling the start-stop service interface and the message topic name, the object name corresponding to the message listening component belonging to the message topic name is determined; the message listening component to be started or stopped is determined according to the object name, and the start component in the message listening component to be started is executed, or the stop component in the message listening component to be stopped is executed.

[0108] The message listening components in the message queue can be divided according to the message topic name (Topic), where the object names of the message listening components divided into the same message topic name belong to the message topic name. On the front-end visualization page or configuration center, you can configure the target message topic name, and through the target message topic name, you can determine the object name belonging to the target message topic name. Through the object name, you can determine the corresponding message listening component to be started or to be stopped in the message queue.

[0109] In the message queue, each message listening component includes a start component and a stop component. If the type indication information indicates starting or stopping the message listening component under the message topic name, then the service interface corresponding to the type indication information is the start-stop service interface.

[0110] Furthermore, the message topic name is obtained from the message reading configuration parameters, and the object name corresponding to the message listening component belonging to the message topic name is determined from the message queue by calling the start-stop service interface and the message topic name; the message listening component to be started or the message listening component to be stopped is located according to the object name. The start component in the message listening component to be started is executed to start the message listening component to be started; or the stop component in the message listening component to be stopped is executed to stop the message listening component to be stopped.

[0111] Taking Kafka as a message queue, each consumer listening container instance includes a start method and a stop method. The start method belongs to the start component, and the stop method belongs to the stop component. If the type indication message indicates to start or stop the consumer listening container instance under the message topic name, then the service interface corresponding to the type indication message is the start-stop service interface.

[0112] Get the message topic name from the message reading configuration parameters. The corresponding object name under the message topic name can be determined through the message topic name. Then locate the consumer listening container instance to be started or the consumer listening container instance to be stopped according to the object name, execute the start method of the consumer listening container instance to be started, and start the consumer listening container instance to be started in Kafka; or execute the stop method of the consumer listening container instance to be stopped, and stop the consumer listening container instance to be stopped in Kafka.

[0113] Based on the aforementioned abstract class of Kafka service and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When executing the start method or stop method of the consumer listening container instance, the start method or stop method of the consumer listening container instance is written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service, and the message topic name is also written into the Kafka tuning method, so as to implement the abstract class of Kafka service and rewrite the Kafka tuning method, so as to start the consumer listening container instance to be started in Kafka or stop the consumer listening container instance to be stopped.

[0114] This embodiment can determine the object name belonging to the message subject name based on the message subject name, and then determine the message listening component to be started that needs to be started, or the message listening component to be stopped that needs to be stopped from the message queue through the object name, so that the message listening component to be started can be started or the message listening component to be stopped can be stopped without restarting the message queue.

[0115] In an embodiment, the service interface includes a parameter adjustment service interface;

[0116] The message reading configuration parameters also include a first parameter and a first parameter value of the first parameter;

[0117] Wherein, determining the service interface according to the type indication information includes:

[0118] If the type indication information indicates that the parameters of the message monitoring component are to be adjusted, determining that the service interface is a parameter adjustment service interface;

[0119] Among them, by calling the service interface, adjusting the message monitoring component includes:

[0120] By calling the parameter adjustment service interface, the first parameter and the first parameter value, a message monitoring component of the message queue is obtained, and the parameter value of the first parameter in the message monitoring component is updated to the first parameter value.

[0121] Another way to adjust the processing capacity of the message queue is to adjust the parameters of the message listening component to adapt to the changing business volume. For example, when encountering a business peak, adjusting the parameters of the message listening component can improve the performance of the message listening component, thereby improving the overall processing capacity of the message queue.

[0122] In the message queue, each message listening component has its own attribute parameters, which determine the performance of the message listening component when facing different types of messages. By adjusting the attribute parameters, the processing ability of the message queue when facing different types of messages can be improved. If the type indication information indicates that the parameters of the message listening component are adjusted, then the service interface corresponding to the type indication information is the parameter adjustment service interface. In this embodiment, to adjust the parameters of all message listening components in the message queue, it is first necessary to read the configuration parameters from the message to obtain the first parameter and the first parameter value of the first parameter; the first parameter is the parameter that needs to be modified in the attribute parameters of the message listening component. The first parameter value is the parameter value that needs to be modified for the first parameter.

[0123] Based on the obtained first parameter and the first parameter value of the first parameter, all message listening components of the message queue are obtained by calling the parameter adjustment service interface; the parameter value of the first parameter of each message listening component in all the message listening components is updated to the first parameter value, so as to realize the modification of the attribute parameters of the message listening component and take effect immediately.

[0124] Taking Kafka as a message queue as an example, the input parameter of the consumer listening container instance is the listening container parameter (containerProperties). If you need to adjust the parameters of the consumer listening container instance, you need to obtain the listening container parameters of the consumer listening container instance; there are many types of listening container parameters for consumer listening container instances. For example, one listening container parameter is "max-poll-records", which represents the number of message data that the Kafka client pulls from the Kafka service at a time. By adjusting this parameter, Kafka's consumption performance can be improved. Another example is a listening container parameter "enable-auto-commit", which indicates whether the Kafka client starts consuming from the earliest offset or the latest offset. By adjusting this parameter, different business scenarios can be met. Among them, offset refers to the position of the next message to be consumed in the partition.

[0125] If the type indication information indicates that the parameters of the consumer listening container instance are to be adjusted, then the service interface corresponding to the type indication information is the parameter adjustment service interface. The first parameter and the first parameter value of the first parameter are obtained from the message reading configuration parameters; by calling the parameter adjustment service interface and the obtained first parameter and the first parameter value of the first parameter, all the consumer listening container instances of Kafka are obtained, and the parameter value of the first parameter in the listening container parameters of the consumer listening container instance is updated to the first parameter value, thereby adjusting the listening container parameters of the consumer listening container instance and taking effect immediately. The consumer listening container instance continues to execute message consumption according to the adjusted listening container parameters that take effect immediately. The adjustment of the listening container parameters of the consumer listening container instance and taking effect immediately is to modify the listening container parameters of the consumer listening container instance through reflection, so that the consumer listening container parameters take effect immediately and dynamically, thereby improving the processing capacity of the message queue.

[0126] Based on the aforementioned abstract class of Kafka service and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When adjusting the parameters of the consumer listening container instance, the parameter adjustment service interface and the first parameter and the first parameter value are written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service to implement the abstract class of Kafka service and rewrite the Kafka tuning method, so as to adjust the parameter value of the first parameter of all consumer listening container instances in Kafka to the first parameter value and take effect immediately.

[0127] This embodiment can adjust the parameter value of the first parameter to the first parameter value and take effect immediately without restarting all message listening components in the message queue according to the type indication information, thereby improving the performance of the message listening component and avoiding message backlog.

[0128] In an embodiment, the service interface includes a parameter adjustment service interface;

[0129] The message reading configuration parameters also include an object name corresponding to the message listening component, a second parameter, and a second parameter value of the second parameter;

[0130] Wherein, determining the service interface according to the type indication information includes:

[0131] If the type indication information indicates that the parameters of the message monitoring component are to be adjusted, determining that the service interface is a parameter adjustment service interface;

[0132] Among them, by calling the service interface, adjusting the message monitoring component includes:

[0133] The message listening component to be adjusted is determined by calling the parameter adjustment service interface and the object name corresponding to the message listening component, and the parameter value of the second parameter in the message listening component to be adjusted is updated to the second parameter value.

[0134] In the message queue, each message listening component has its own attribute parameters, which determine the performance of the message listening component when facing different types of messages. By adjusting the attribute parameters, the processing ability of the message queue when facing different types of messages can be improved. If the type indication information indicates that the parameters of one or some message listening components are to be adjusted, then the service interface corresponding to the type indication information is the parameter adjustment service interface. In this embodiment, to adjust the parameters of some message listening components in the message queue, it is first necessary to read the configuration parameters from the message to obtain the object name corresponding to the message listening component, as well as the second parameter and the second parameter value of the second parameter. The object name is used to locate the message listening component that needs to be adjusted. The second parameter is the parameter that needs to be modified in the attribute parameters of the message listening component. The second parameter value is the parameter value that needs to be modified for the second parameter.

[0135] By calling the parameter adjustment service interface and the object name corresponding to the message listening component, the message listening component whose parameters are to be adjusted is located from the message queue, and the parameter value of the second parameter in the message listening component whose parameters are to be adjusted is updated to the second parameter value, so as to realize the modification of the attribute parameters of the message listening component and take effect immediately.

[0136] Taking Kafka as a message queue as an example, if the type indication information indicates that the parameters of some consumer listening container instances are adjusted, then the service interface corresponding to the type indication information is a parameter adjustment service interface. The object name corresponding to the consumer listening container instance, the second parameter, and the second parameter value of the second parameter are obtained from the message reading configuration parameters; by calling the parameter adjustment service interface and the object name corresponding to the consumer listening container instance, the consumer listening container instance whose parameters are to be adjusted is located from Kafka, and the parameter value of the second parameter in the listening container parameter of the consumer listening container instance whose parameters are to be adjusted is updated to the second parameter value, thereby realizing the adjustment of the listening container parameter of the consumer listening container instance whose parameters are to be adjusted and taking effect immediately. The consumer listening container instance whose parameters are to be adjusted continues to execute message consumption according to the adjusted listening container parameters that take effect immediately. The adjustment of the listening container parameters of the consumer listening container instance whose parameters are to be adjusted and taking effect immediately is to modify the listening container parameters of the consumer listening container instance through reflection, so that the consumer listening container parameters take effect immediately and dynamically, thereby realizing the improvement of the processing capacity of the message queue.

[0137] Based on the aforementioned abstract class of Kafka service, and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When adjusting the parameters of the consumer listening container instance, the parameter adjustment service interface and the object name corresponding to the consumer listening container instance, as well as the second parameter and the second parameter value are written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service to implement the abstract class of Kafka service and rewrite the Kafka tuning method, so as to adjust the parameter value of the second parameter of the consumer listening container instance whose parameter in Kafka is to be adjusted to the second parameter value and take effect immediately.

[0138] This embodiment can determine the message listening component whose parameters need to be adjusted from the message queue based on the object name corresponding to the message listening component, so that the parameter value of the second parameter of the message listening component whose parameters need to be adjusted is updated to the second parameter value without restarting the message queue and the update takes effect immediately, thereby improving the performance of the message listening component and avoiding message backlogs.

[0139] In an embodiment, the service interface includes a parameter adjustment service interface;

[0140] The message reading configuration parameters also include a message topic name, a third parameter, and a third parameter value of the third parameter;

[0141] Wherein, determining the service interface according to the type indication information includes:

[0142] If the type indication information indicates that the parameters of the message monitoring component are to be adjusted, determining that the service interface is a parameter adjustment service interface;

[0143] Among them, by calling the service interface, adjusting the message monitoring component includes:

[0144] By calling the parameter adjustment service interface and the message topic name, the object name of the message listening component to be adjusted is determined, the message listening component to be adjusted is determined according to the object name, and the parameter value of the third parameter in the message listening component to be adjusted is updated to the third parameter value.

[0145] The message listening components in the message queue can be divided according to the message topic name (Topic), where the object names of the message listening components divided into the same message topic name belong to the message topic name. On the front-end visualization page or configuration center, you can configure the target message topic name, and through the target message topic name, you can determine the object name belonging to the target message topic name. Through the object name, you can determine the corresponding message listening component whose parameters are to be adjusted in the message queue.

[0146] In the message queue, each message listening component has its own attribute parameters, which determine the performance of the message listening component when facing different types of messages. By adjusting the attribute parameters, the processing ability of the message queue when facing different types of messages can be improved. If the type indication information indicates that the message listening component under the message subject name is to be adjusted, then the service interface corresponding to the type indication information is the parameter adjustment service interface. In this embodiment, to adjust the parameters of the message listening components under the same message subject name in the message queue, it is first necessary to read the configuration parameters from the message to obtain the message subject name, the third parameter and the third parameter value of the third parameter. The message subject name is used to determine the object name of the message listening component whose parameters are to be adjusted belonging to the message subject name; the object name of the message listening component whose parameters are to be adjusted is used to locate the message listening component whose parameters are to be adjusted from the message queue. The third parameter is the parameter that needs to be modified in the attribute parameters of the message listening component. The third parameter value is the parameter value that needs to be modified for the third parameter.

[0147] By calling the parameter adjustment service interface and the message topic name, the object name of the message listening component whose parameters are to be adjusted belonging to the message topic name is determined, and the message listening component whose parameters are to be adjusted is further located from the message queue according to the object name, and the parameter value of the third parameter in the message listening component whose parameters are to be adjusted is updated to the third parameter value, so as to realize the modification of the attribute parameters of the message listening component and make it take effect immediately.

[0148] Taking Kafka as a message queue as an example, if the type indication information indicates that the parameters of the consumer listening container instance belonging to the message topic name are adjusted, then the service interface corresponding to the type indication information is the parameter adjustment service interface. The message topic name, the third parameter and the third parameter value of the third parameter are obtained from the message reading configuration parameters; by calling the parameter adjustment service interface and the message topic name, the parameter belonging to the message topic name is to be adjusted, and the consumer listening container instance whose parameter is to be adjusted is located from Kafka, and the parameter value of the third parameter in the listening container parameter of the consumer listening container instance whose parameter is to be adjusted is updated to the third parameter value, thereby realizing the adjustment of the listening container parameter of the consumer listening container instance whose parameter is to be adjusted and taking effect immediately. The consumer listening container instance whose parameter is to be adjusted continues to execute message consumption according to the adjusted listening container parameter that takes effect immediately. The adjustment of the listening container parameter of the consumer listening container instance whose parameter is to be adjusted and taking effect immediately is to modify the listening container parameter of the consumer listening container instance through reflection, so that the consumer listening container parameter takes effect immediately and dynamically, thereby realizing the improvement of the processing capacity of the message queue.

[0149] Based on the aforementioned abstract class of Kafka service, and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When adjusting the parameters of the consumer listening container instance, the parameter adjustment service interface and the message topic name as well as the third parameter and the third parameter value are written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service to implement the abstract class of Kafka service and rewrite the Kafka tuning method, so as to adjust the parameter value of the third parameter of the consumer listening container instance whose parameter in Kafka is to be adjusted to the third parameter value and take effect immediately.

[0150] This embodiment can determine the object name belonging to the message topic name based on the message topic name, and then determine the message listening component whose parameters need to be adjusted from the message queue through the object name, so that the parameter value of the third parameter of the message listening component whose parameters need to be adjusted is updated to the third parameter value without restarting the message queue and the update takes effect immediately, thereby improving the performance of the message listening component and avoiding message backlogs.

[0151] In the embodiment, the service interface includes a new service interface;

[0152] Wherein, determining the service interface according to the type indication information includes:

[0153] If the type indication information indicates that a message listening component is to be created in the message queue, determining that the service interface is a newly added service interface;

[0154] Wherein, by calling the service interface, a message monitoring component is created in the message queue, including:

[0155] By calling the new service interface, a new message listening component is created for the message queue, and the parameters of the new message listening component are configured as the first message processing parameters.

[0156] Another way to adjust the processing capacity of a message queue is to create a new message listening component in the message queue. By increasing the number of message listening components in the message queue, the overall processing capacity of the message queue can be improved.

[0157] There are already multiple message listening components running in the message queue. If the existing multiple message listening components are not enough to cope with the business peak, then it is necessary to create a new message listening component in the message queue. Specifically, if the type indication information indicates to create a message listening component in the message queue, then the service interface corresponding to the type indication information is a newly added service interface. By calling the newly added service interface, a new message listening component is created for the message queue. Among them, the newly created message listening component is in the initial state. If you want to deal with the message being processed, you need to configure the parameters of the new message listening component as the first message processing parameter. The first message processing parameter is the attribute parameter of the existing message listening component in the message queue. The first message processing parameter is obtained by obtaining the attribute parameter of the existing message listening component in the message queue. After configuring the first message processing parameter for the newly created message listening component, the startup component is executed for the newly created message listening component, and the newly created message listening component has been started to execute message consumption.

[0158] Taking Kafka as a message queue as an example, each consumer listening container instance in Kafka has its own attribute parameters. The attribute parameters of the consumer listening container instance are obtained to determine the first message processing parameters.

[0159] If the type indication information indicates to create a consumer listening container instance in Kafka, the service interface corresponding to the type indication information is a newly added service interface;

[0160] Call the newly added service interface to create a new consumer listening container instance for Kafka, configure the parameters of the newly created consumer listening container instance as the first message processing parameters, and execute the startup method on the newly created consumer listening container instance to start the newly created consumer listening container instance to execute message consumption.

[0161] Based on the aforementioned abstract class of Kafka service, and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When a message listening component is created in the message queue, the processing logic of creating the message listening component in the message queue is written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service, and the first message processing parameter is also written into the Kafka tuning method to implement the abstract class of Kafka service and rewrite the Kafka tuning method, thereby creating a new consumer listening container instance in Kafka and starting it.

[0162] This embodiment obtains the attribute parameters of the existing message listening component in the message queue as the first message processing parameters, creates an information message listening component in the message queue by calling the new service interface, and configures the parameters of the message listening component as the first message processing parameters, thereby expanding the capacity of the message listening component in the message queue, thereby improving the processing capability of the message queue.

[0163] In the embodiment, the service interface includes a new service interface;

[0164] The message reading configuration parameters also include second message processing parameters;

[0165] Wherein, determining the service interface according to the type indication information includes:

[0166] If the type indication information indicates that a message listening component is to be created in the message queue, determining that the service interface is a newly added service interface;

[0167] Among them, by calling the service interface, a message listening component is created in the message queue, including:

[0168] By calling the new service interface, a new message listening component is created in the message queue, and the parameters of the new message listening component are configured as the second message processing parameters.

[0169] If the type indication information indicates that a message listening component is to be created in the message queue, then the service interface corresponding to the type indication information is a newly added service interface. By calling the newly added service interface, a new message listening component is created for the message queue. Among them, the newly created message listening component is in the initial state. If you want to deal with the message being processed, you need to configure the parameters of the new message listening component as the second message processing parameters. The second message processing parameters are the default attribute parameters pre-configured in the message queue. The second message processing parameters are the default attribute parameters pre-configured according to the historical operating parameters before the message queue runs. After configuring the second message processing parameters for the newly created message listening component, the startup component is executed for the newly created message listening component, the newly created message listening component has been started, and message consumption is executed.

[0170] Taking Kafka as a message queue as an example, the default attribute parameters are pre-configured in Kafka, the pre-configured default attribute parameters are obtained, and the second message processing parameters are determined.

[0171] If the type indication information indicates to create a consumer listening container instance in Kafka, then the service interface corresponding to the type indication information is a newly added service interface; the newly added service interface is called to create a new consumer listening container instance for Kafka, the parameters of the newly created consumer listening container instance are configured as the second message processing parameters, and the startup method is executed on the newly created consumer listening container instance to start the newly created consumer listening container instance to perform message consumption.

[0172] Based on the aforementioned abstract class of Kafka service, and the subclass of the abstract class of Kafka service that inherits from the abstract class of Kafka service, the Kafka tuning method provided by the abstract class of Kafka service is a public method. When a message listening component is created in the message queue, the processing logic of creating the message listening component in the message queue is written into the Kafka tuning method as the processing logic of the subclass of the abstract class of Kafka service, and the second message processing parameter is also written into the Kafka tuning method to implement the abstract class of Kafka service and rewrite the Kafka tuning method, thereby creating a new consumer listening container instance in Kafka and starting it.

[0173] This embodiment uses the default attribute parameters pre-configured in Kafka as the second message processing parameters, creates an information message listening component in the message queue by calling the new service interface, and configures the parameters of the message listening component as the second message processing parameters to achieve the expansion of the message listening component in the message queue, thereby improving the processing capability of the message queue.

[0174] In the embodiment, the service interface includes a newly added service interface;

[0175] The message reading configuration parameters also include quantity;

[0176] Wherein, determining the service interface according to the type indication information includes:

[0177] If the type indication information indicates that a message listening component is to be created in the message queue, determining that the service interface is a newly added service interface;

[0178] Among them, by calling the service interface, a message listening component is created in the message queue, including:

[0179] Through the newly added service interface, a corresponding number of new message listening components are created in the message queue.

[0180] If you need to create multiple message listening components in a message queue through one operation, you also need to obtain the number of message listening components to be created from the message reading configuration parameters. Specifically, if the type indication information indicates to create a message listening component in the message queue, the service interface corresponding to the type indication information is a new service interface. By calling the new service interface, a corresponding number of new message listening components are created for the message queue.

[0181] Taking Kafka as a message queue, if you need to create a set number of consumer listening container instances in Kafka in one operation, you first need to determine the type indication information indicating the creation of consumer listening container instances in Kafka. Then the service interface corresponding to the type indication information is a new service interface; call the new service interface to create a corresponding number of consumer listening container instances for Kafka. And execute the startup method for the corresponding number of newly created consumer listening container instances to start the corresponding number of newly created consumer listening container instances to perform message consumption.

[0182] The disclosed embodiment can create a corresponding number of message listening components through a single adding operation, thereby realizing batch creation of message listening components. In response to a surge in business, multiple message listening components can be created with one click, thereby coping with the surge in business.

[0183] The consumption sequence of the consumer listening container instance of the Kafka service is a parallel sequence in one case. For example, there are five consumer listening container instances in the Kafka service. These five consumer listening container instances consume messages in parallel. When the expanded consumer listening container instance is added to the consumption sequence of the new consumer listening container instance, five expanded consumer listening container instances are added in parallel on the basis of the original five parallel consumer listening container instances. At this time, the Kafka service has ten consumer listening container instances, which can greatly improve the consumption capacity of the Kafka service.

[0184] Figure 4 The architecture diagram of the embodiment of the present disclosure is shown as follows: Figure 4 As shown, the embodiment of the present disclosure provides a method for efficiently completing the start and stop management, parameter adjustment and capacity expansion of Kafka through a visual configuration page, thereby improving the consumption capacity of Kafka.

[0185] A front-end visualization page is constructed on the front end, a Kafka configuration scenario option 41 is set in the page, a selection operation for the Kafka configuration scenario is received, and in response to the selected Kafka configuration scenario option, the type indication information corresponding to the selected Kafka configuration scenario option is determined as a message reading configuration parameter, and the message reading configuration parameter is sent to the Kafka tuning processor kafkaHandler 42 at the back end through the transmission channel established by HTTP, and an abstract class is implemented in the Kafka tuning processor, respectively for executing a message listening component 431, adjusting a message listening component 432, and creating a message listening component 433 in a message queue.

[0186] When the message monitoring component is executed, the consumer monitoring container instance is traversed, the start method or stop method of the consumer monitoring container instance is obtained by calling the start and stop service interface, and the start method or stop method is executed to realize the dynamic management of Kafka start and stop 441.

[0187] When adjusting the message listening component, the target parameter and the target parameter value of the target parameter are obtained from the message reading configuration parameter; wherein the target parameter can be the first parameter, the second parameter or the third parameter, and the target parameter value can be the first parameter value, the second parameter value or the third parameter value. By calling the parameter adjustment service interface, the parameter value of the target parameter of the consumer listening container instance is modified to the target parameter value, thereby realizing dynamic modification of Kafka consumption parameters to improve consumption capacity 442.

[0188] When creating a message listening component in the message queue, first determine the message processing parameter, which can be the first message processing parameter or the second message processing parameter. By calling the new service interface, create a new consumer listening container instance for Kafka, and configure the message processing parameters for the new consumer listening container instance, so as to increase the number of consumers and improve the consumption capacity 443 by adding a new consumer listening container as a consumer.

[0189] The adjustment of Kafka's message processing capacity was completed by implementing dynamic management of Kafka's start and stop, dynamically modifying Kafka's consumption parameters to improve consumption capacity, increasing the number of consumer listening container instances, and adding new consumer listening containers as consumers to increase the number of consumers and improve consumption capacity.

[0190] The disclosure can better cope with special scenarios, especially when the burst traffic is relatively high and the consumption capacity needs to be improved to avoid message accumulation. The start and stop of Kafka consumption can be dynamically controlled according to business needs. At the same time, in order to cope with the scenario where the consumption capacity is insufficient due to the sudden increase in traffic during the peak of sudden business, it can support dynamic adjustment of parameters to improve Kafka's consumption capacity; according to this solution combined with the development of the front-end page, a visual configuration page can be provided for operation and maintenance personnel to configure and take effect, and to maintain and deal with sudden scenarios; adjusting parameters does not require restarting online services, which minimizes the impact.

[0191] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.

[0192] Based on the same inventive concept, the present disclosure also provides a data processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0193] Figure 5 A schematic diagram of a data processing device in an embodiment of the present disclosure is shown. Figure 5 As shown, the device comprises:

[0194] A message reading configuration parameter acquisition module 501 is used to acquire a message reading configuration parameter, wherein the message reading configuration parameter includes type indication information;

[0195] A service interface determination module 502, configured to determine a service interface according to the type indication information;

[0196] The message monitoring component processing module 503 is used to process the message monitoring component of the message queue by calling the service interface.

[0197] It should be noted that the message reading configuration parameter acquisition module 501, the service interface determination module 502 and the message monitoring component processing module 503 correspond to S202 to S206 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0198] The present disclosure also provides a data processing method, which is applied to a front-end visualization page, and the specific steps include:

[0199] Display the configuration page;

[0200] Configuring message reading configuration parameters in the configuration page, wherein the message reading configuration parameters include type indication information, and the type indication information is used to determine the service interface;

[0201] The message is sent to read the configuration parameters to call the service interface and process the message monitoring component of the message queue.

[0202] Based on the same inventive concept, the present disclosure also provides a data processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0203] The present disclosure provides a schematic diagram of a data processing device, the device comprising:

[0204] Configuration page display module, used to display the configuration page;

[0205] A configuration parameter reading module, used to configure message reading configuration parameters in the configuration page, wherein the message reading configuration parameters include type indication information, and the type indication information is used to determine a service interface;

[0206] The message reading configuration parameter sending module is used to send the message reading configuration parameters to call the service interface and process the message monitoring component of the message queue.

[0207] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0208] Reference below Figure 6 , the electronic device is exemplified in the form of a general-purpose computing device. It should be understood that Figure 6 The electronic device 60 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0209] like Figure 6As shown, the electronic device 60 includes: at least one processor 601; at least one memory 602, and one or more I / O interfaces 603, connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs that can be executed by at least one processor 601, and the one or more computer programs are executed by at least one processor 601 so that the at least one processor 601 can execute the above-mentioned method of searching for web pages.

[0210] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0211] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0212] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned data processing method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0213] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0214] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above-mentioned data processing method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0215] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0216] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0217] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0218] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0219] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0220] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0221] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0222] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A data processing method, characterized in that: include: Acquire message reading configuration parameters, wherein the message reading configuration parameters include type indication information; Determining a service interface according to the type indication information; By calling the service interface, the message monitoring component of the message queue is processed.

2. The method according to claim 1, characterized in that By calling the service interface, the message monitoring component of the message queue is processed, including: By calling the service interface, the message monitoring component is executed, or the message monitoring component is adjusted, or a message monitoring component is created in the message queue.

3. The method according to claim 2, characterized in that The message monitoring component includes a start component and a stop component; the service interface includes a start and stop service interface; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates starting or stopping the message monitoring component, determining that the service interface is the start-stop service interface; The executing of the message monitoring component by calling the service interface includes: The message monitoring component of the message queue is obtained by calling the start-stop service interface, and the start component or stop component in the message monitoring component is executed.

4. The method according to claim 2, characterized in that: The message monitoring component includes a start component and a stop component; the service interface includes a start and stop service interface; The message reading configuration parameters also include the object name corresponding to the message monitoring component; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates starting or stopping the message monitoring component, determining that the service interface is the start-stop service interface; The executing of the message monitoring component by calling the service interface includes: By calling the start-stop service interface and the object name corresponding to the message listening component, the message listening component to be started or stopped is determined, and the start component in the message listening component to be started is executed, or the stop component in the message listening component to be stopped is executed.

5. The method according to claim 2, characterized in that: The message monitoring component includes a start component and a stop component; the service interface includes a start and stop service interface; The message reading configuration parameters also include a message topic name; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates starting or stopping the message monitoring component, determining that the service interface is a start-stop service interface; The executing of the message monitoring component by calling the service interface includes: By calling the start-stop service interface and the message topic name, the object name corresponding to the message listening component belonging to the message topic name is determined; the message listening component to be started or stopped is determined according to the object name, and the start component in the message listening component to be started is executed, or the stop component in the message listening component to be stopped is executed.

6. The method according to claim 2, characterized in that The service interface includes a parameter adjustment service interface; The message reading configuration parameters also include a first parameter and a first parameter value of the first parameter; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates that the parameters of the message monitoring component are to be adjusted, determining that the service interface is the parameter adjustment service interface; The adjusting the message monitoring component by calling the service interface includes: The message monitoring component of the message queue is obtained by calling the parameter adjustment service interface, the first parameter and the first parameter value, and the parameter value of the first parameter in the message monitoring component is updated to the first parameter value.

7. The method according to claim 2, characterized in that: The service interface includes a parameter adjustment service interface; The message reading configuration parameters also include an object name corresponding to the message listening component, a second parameter, and a second parameter value of the second parameter; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates that the parameters of the message monitoring component are to be adjusted, determining that the service interface is the parameter adjustment service interface; The adjusting the message monitoring component by calling the service interface includes: The message listening component whose parameter is to be adjusted is determined by calling the parameter adjustment service interface and the object name corresponding to the message listening component, and the parameter value of the second parameter in the message listening component whose parameter is to be adjusted is updated to the second parameter value.

8. The method according to claim 2, characterized in that: The service interface includes a parameter adjustment service interface; The message reading configuration parameters also include a message subject name, a third parameter and a third parameter value of the third parameter; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates that the parameters of the message monitoring component are to be adjusted, determining that the service interface is the parameter adjustment service interface; The adjusting the message monitoring component by calling the service interface includes: By calling the parameter adjustment service interface and the message topic name, the object name of the message listening component to be adjusted is determined, the message listening component to be adjusted is determined according to the object name, and the parameter value of the third parameter in the message listening component to be adjusted is updated to the third parameter value.

9. The method according to claim 2, characterized in that: The service interface includes a new service interface; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates to create a message listening component in the message queue, determining that the service interface is the newly added service interface; The step of creating a message monitoring component in the message queue by calling the service interface includes: By calling the newly added service interface, a new message monitoring component is created for the message queue, and the parameters of the new message monitoring component are configured as the first message processing parameters.

10. The method according to claim 2, characterized in that The service interface includes a new service interface; The message reading configuration parameters also include second message processing parameters; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates to create a message listening component in the message queue, determining that the service interface is the newly added service interface; The step of creating a message monitoring component in the message queue by calling the service interface includes: By calling the newly added service interface, a new message listening component is created in the message queue, and the parameters of the new message listening component are configured as the second message processing parameters.

11. The method according to claim 2, characterized in that The service interface includes a newly added service interface; The message reading configuration parameters also include quantity; Wherein, determining the service interface according to the type indication information includes: If the type indication information indicates to create a message listening component in the message queue, determining that the service interface is the newly added service interface; The step of creating a message monitoring component in the message queue by calling the service interface includes: Through the newly added service interface, a corresponding number of newly added message listening components are created in the message queue.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1 to 11 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images