Message processing method and related device
Patent Information
- Application Number
- CN202411720368.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-04-08
Smart Images
Figure CN119781999A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of computer data processing, and in particular to a message processing method and related devices. Background Art
[0002] In the message processing process, if the consumption speed of message consumers cannot keep up with the sending speed of message producers, more and more unprocessed messages will be generated, and these messages are called accumulated messages. Message accumulation will cause delays in message consumption.
[0003] At present, the consumption speed is generally improved through emergency capacity expansion, adding consumers, and increasing consumer consumption efficiency. However, when a large number of messages accumulate, the newly generated messages will be delayed in consumption and cannot meet business needs. Summary of the invention
[0004] In view of this, the present application provides a message processing method and related devices, which can greatly alleviate message accumulation while processing newly generated messages in a timely manner to avoid message delays.
[0005] In a first aspect, an embodiment of the present application provides a message processing method, the method comprising:
[0006] When messages in a message partition are piled up, obtaining a first consumption displacement of a first consumer of the message partition and an end displacement of the message partition, wherein the first consumption displacement is position information of messages that have been consumed by the first consumer;
[0007] In case the first consumption displacement is adjusted to the end displacement, the first consumer consumes messages from the end displacement;
[0008] Determine a second consumer of the message partition, and the second consumer starts consuming messages from the first consumption shift;
[0009] The message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is one digit before the end displacement, the second consumer stops consuming messages.
[0010] In a second aspect, an embodiment of the present application provides a message processing device, the device comprising:
[0011] an acquiring unit, configured to acquire, when messages in a message partition are piled up, a first consumption displacement of a first consumer of the message partition and an end displacement of the message partition, wherein the first consumption displacement is position information of messages that have been consumed by the first consumer;
[0012] A first adjusting unit, configured to cause the first consumer to consume messages from the terminal displacement when the first consumption displacement is adjusted to the terminal displacement;
[0013] A second adjustment unit is used to determine a second consumer of the message partition, and the second consumer starts consuming messages from the first consumption displacement;
[0014] The consumption control unit is used to monitor the message displacement of the second consumer. If the consumption displacement of the second consumer is one digit before the end displacement, the second consumer stops consuming messages.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps of any method in the first aspect of the embodiment of the present application.
[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in any method of the first aspect of the embodiment of the present application.
[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product may be a software installation package.
[0018] It can be seen that through the above-mentioned message processing method and related devices, first, when the messages in the message partition are piled up, the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition are obtained, and the first consumption displacement is the position information of the message that the first consumer has consumed; when the first consumption displacement is adjusted to the end displacement, the first consumer consumes the message from the end displacement; the second consumer of the message partition is determined, and the second consumer starts to consume the message from the first consumption displacement; the message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is the previous position of the end displacement, the second consumer stops consuming the message. In the case of message accumulation, the accumulated messages and the newly generated messages can be isolated and processed in parallel, which greatly alleviates the message accumulation and can process the newly generated messages in time to avoid message delays. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0020] Figure 1 A system architecture diagram of a message processing method provided in an embodiment of the present application;
[0021] Figure 2 A schematic diagram of consumption displacement of a message partition provided in an embodiment of the present application;
[0022] Figure 3 A flowchart of a message processing method provided in an embodiment of the present application;
[0023] Figure 4 A flowchart of another message processing method provided in an embodiment of the present application;
[0024] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0025] Figure 6 A block diagram of the functional units of a message processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0027] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0028] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship. The "plurality" appearing in the embodiments of the present application refers to two or more.
[0029] In the embodiments of the present application, "at least one item" or similar expressions refer to any combination of these items, including any combination of single items or plural items, and refer to one or more, and multiple refers to two or more. For example, at least one item of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Among them, each of a, b, and c can be an element or a set containing one or more elements.
[0030] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.
[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0032] The following is an explanation of the relevant contents, concepts, meanings, technical issues, technical solutions, beneficial effects, etc. involved in the embodiments of the present application.
[0033] I. Related terms
[0034] The message in the embodiments of the present application refers to the message in the distributed publish-subscribe message system and the message queue. The message producer sends the generated message to the message queue middleware, and the message consumer pulls the message from the message queue middleware for consumption. The message is stored in an orderly manner. Message queue products include Kafka, ActiveMQ, RabbitMQ, RocketMQ, etc.
[0035] Topic: Messages can be classified into different topics. Each topic can include multiple partitions, and each message will only be saved in one partition.
[0036] Offset indicates the corresponding position of the message in the partition, and can also indicate the position consumed by the consumer.
[0037] A consumer group consists of multiple consumers. Each consumer in a consumer group is responsible for consuming data from different partitions. A partition can only be consumed by consumers in one group. Consumer groups do not affect each other. All consumers belong to a consumer group, that is, a consumer group is a logical subscriber.
[0038] 2. Application scenarios and electronic devices
[0039] 1. Application scenarios
[0040] The embodiments of the present application can be applied to the following application scenarios, including but not limited to: asynchronous processing, application decoupling, traffic clipping and message communication, etc. It should be noted that the embodiments of the present application have a good effect on services with high real-time requirements, such as snapshot recognition and alarm services in the security industry, which are not specifically limited here.
[0041] 2. Electronic equipment
[0042] The electronic device in the embodiment of the present application may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a wearable electronic device with a wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with an IOS system, an Android system, a Microsoft system, or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer (Laptop), etc. It should also be understood that in some other embodiments, the above-mentioned electronic device may not be a portable electronic device, but a desktop computer, a server, etc.
[0043] 3. Example
[0044] See also Figure 1 , Figure 1 This is a system architecture diagram of a message processing method provided in an embodiment of the present application. Figure 1 As shown, it includes a message producer, a message queue middleware, a first consumer 110, a second consumer 120, a message processing trigger service 130, a message processing function 140 and a database.
[0045] There may be multiple first consumers 110 belonging to a first consumer group. It can be understood that the first consumer 110 is a consumer that exists in the current message processing stage.
[0046] There may be multiple second consumers 120 belonging to a second consumer group. It can be understood that the second consumer 120 is a constructed temporary consumer used to consume accumulated messages.
[0047] Among them, the message processing trigger service 130 can be used to determine whether to execute the message processing method of the embodiment of the present application. When the message processing trigger service 130 determines that a partition in any topic is in a message accumulation state, the message processing of the embodiment of the present application can be executed on the partition. Specifically, the message processing trigger service 130 can be a service for centralized management of application configuration, which can separate the configuration of the application from the code and the local configuration file to achieve centralized management and dynamic update of the configuration. For example, the message processing trigger service 130 can be a Nacos configuration center, which is not specifically limited here.
[0048] Among them, the message processing function 140 can be called after the above-mentioned message processing trigger service 130 determines to execute the message processing method on the message partition, and is used to adjust the consumption displacement of the first consumer 110, create the second consumer 120, and set the consumption displacement of the second consumer 120, and can control the consumption process of the first consumer 110 and the second consumer 120. Specifically, the message processing function 140 can be a function method in a Java environment, which can be associated with a consumer listening container instance, and implement the above functions through multiple built-in methods in the consumer listening container instance.
[0049] It should be noted that when consumers consume messages, they can complete the consumption by creating a consumer listening container instance (Message Listener Container) through spring framework scanning, which will not be elaborated here.
[0050] First, the message producer sends the message to each partition of the topic of the message queue middleware for storage, and the first consumer 110 pulls the message from the message queue middleware. When any message partition is in a message accumulation state, the message processing trigger service 130 can determine the identifier of the topic corresponding to the message partition and determine to execute the message processing method of the embodiment of the present application for the message partition. By calling the message processing function 140, the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition are obtained. The first consumption displacement is the position information of the message that the first consumer has consumed. The first consumer 110 starts consuming from the end displacement of the message partition, constructs the second consumer 120 and controls the second consumer to start consuming from the first consumption displacement of the message partition, and stops consuming when consuming to the position before the end displacement of the message partition. The consumed messages will be stored in the database.
[0051] It can be seen that through the above system architecture, when messages are piled up, the accumulated messages and newly generated messages can be isolated and processed in parallel, which greatly alleviates the message accumulation and processes the newly generated messages in time to avoid message delays.
[0052] 3. A message processing method
[0053] At present, the reasons for message accumulation generally include insufficient consumption capacity on the consumer side, that is, when the processing speed on the consumer side cannot keep up with the sending speed on the producer side, it will cause messages to accumulate in the message queue middleware; the number of partitions of the topic in the message queue middleware is insufficient. The number of partitions determines the concurrency of the consumer. Insufficient number of partitions will cause consumers to be unable to fully utilize resources, thus affecting the consumption speed; the message size is too large. Too large a message may cause consumers to spend too long processing a single message, reducing the overall consumption speed; network delay: network delay may cause consumers to get messages from the message queue middleware more slowly. The existing method generally increases the consumption speed by urgently expanding capacity to increase consumers. However, when a large number of messages accumulate, even short-term expansion cannot complete the consumption of messages in time. Messages will still be delayed because the messages in the message queue are sequential. Before the previously accumulated messages are consumed, the newly generated messages will not be consumed in time, resulting in the newly generated messages being delayed until the accumulated messages are consumed. This will affect some businesses with high real-time requirements.
[0054] To solve the above problems, the embodiments of the present application provide a message processing method and related devices, which can isolate the accumulated messages and newly generated messages and process them in parallel when messages are accumulated. This can greatly alleviate the message accumulation and process the newly generated messages in a timely manner to avoid message delays.
[0055] The technical solutions, beneficial effects, concepts, etc. involved in the embodiments of the present application are described in detail below.
[0056] The message processing method of the embodiment of the present application is applied to the scenario where any message partition is in a message accumulation state, and the server is used as the execution subject. First, the message accumulation state is explained. When the speed at which the consumer corresponding to any message partition pulls messages from the topic is less than the speed at which the message producer sends messages to the topic, it can be determined that the topic is in a message accumulation state. At this time, it can be determined which message partitions in the topic are in an accumulation state. Specifically, the message processing method of the embodiment of the present application can be started through a message processing trigger service. The message processing trigger service can be a Nacos configuration center. Two configurations can be provided through the Nacos configuration center. The first configuration is the topic where the message accumulation occurs, and the second configuration is whether to trigger the processing. The server can start a thread to loop and listen to the above configuration. When it is determined that the message accumulation occurs and the processing is triggered, the identification of the topic can be read to determine the message partition in the topic where the message accumulation occurs, and the message processing method of the embodiment of the present application is executed. It can be understood that the message processing method of the embodiment of the present application can be executed on one message partition, two message partitions, or multiple message partitions. The following content is only explained by taking one message partition as an example, and will not be repeated here.
[0057] In order to make the message processing method of the embodiment of the present application clearer, the embodiment of the present application introduces the concept of the first consumer displacement of the message partition. It can be understood that the first consumer displacement of the message partition is the location information of the message that the first consumer of the message partition has consumed, and the first consumer of the message partition is the consumer of the message partition itself. It is named the first consumer here to distinguish it from the subsequent second consumer.
[0058] Among them, the embodiment of the present application also introduces the concept of the end displacement of the message partition. It can be understood that the topic includes multiple message partitions, and the displacement of the last digit of each message partition is the end displacement of the message partition.
[0059] Among them, if the messages in the message partition are in a pile-up state, the first consumption displacement of the first consumer of the message partition can be obtained by calling the first method in the consumer listening container instance; the end displacement of the message partition can be obtained by calling the second method in the consumer listening container instance. The creation of the consumer listening container instance can be performed by the springboot framework scanning the annotation of the consumer method. It can be understood that the consumer listening container instance can be used to consume messages. Specifically, the first method can be a position query function such as the position method, which can be used to query the first consumption displacement of each message partition in the topic, and the second method can be an end position query function such as the endOffsets method, which is used to query the end displacement of each message partition in the topic. It can be understood that at this time, the first consumer of the message partition has consumed the first consumption displacement of the message partition at the position where it consumes the message in its corresponding message partition. At this time, new messages are still being sent to the message queue middleware, and the first consumer of the message partition cannot consume the latest message in time.
[0060] In a possible embodiment, the situation in which messages in a message partition are piled up may be a situation in which a difference between a speed of newly adding messages and a speed of consuming messages in the message partition is greater than a speed threshold.
[0061] In this way, data support can be provided for the subsequent adjustment of the message consumption order.
[0062] Among them, in order to alleviate message accumulation, the consumption displacement of the first consumer of the message partition can be adjusted to the end displacement of each partition. The consumption displacement of the first consumer of the message partition can be set to the end displacement of the message partition by calling the third method of the consumer listening container instance, so that the first consumer of the message partition starts consuming messages from the end displacement of the message partition. The third method can be a search function such as a seek method, which is used to specify that the first consumer of the message partition starts consuming messages from the end displacement of the message partition.
[0063] In this way, the first consumer of the message partition can directly start consuming the latest message, preventing message delays and greatly improving the quality of business processing.
[0064] For accumulated messages, an embodiment of the present application introduces the concept of a second consumer, wherein the second consumer is a constructed temporary consumer. A second consumer group can be constructed by setting consumer parameters, wherein the second consumer group includes the second consumer, and then subscribes to the topic. Finally, by calling a third method of the consumer listening container instance, the consumption displacement of the second consumer of the message partition is set to the first consumption displacement of the message partition, so that the second consumer of the message partition starts consuming messages from the first consumption displacement of the message partition. Similarly, the third method can be a search function such as a seek method, which is used to specify that the second consumer of a message partition starts consuming from the first consumption displacement of the message partition.
[0065] Specifically, a new consumer, i.e., the second consumer, can be created through the consumer interface. For ease of understanding, Kafka is used as an example for explanation. The pseudo code is as follows:
[0066]
[0067]
[0068] It can be seen that by constructing a new consumer and binding it to the second consumer group, and setting the consumption displacement of the second consumer of the message partition to the first consumption displacement of the message partition through the seek method, the second consumer of the message partition can start consuming messages from the first consumption displacement of the message partition. In this way, the accumulated messages can be processed while ensuring that the latest messages will not be delayed.
[0069] Furthermore, if the consumption displacement of the second consumer of the message partition reaches the position before the end displacement of the message partition, the second consumer of each partition is controlled to stop consumption. The consumption displacement of the second consumer of the message partition can be monitored, and if it is detected that the consumption displacement of the second consumer of each partition is the position before the end displacement of each partition, the second consumer group is unsubscribed from the topic to control the second consumer of the message partition to stop consumption.
[0070] Specifically, taking Kafka as an example, the pseudo code is as follows:
[0071]
[0072] It can be seen that offsetB is the end displacement of the message partition. If the second consumer of the message partition consumes to offsetB-1, the subscription to the topic will be cancelled. This is because the first consumer of the message partition has started consuming from the end displacement of the message partition. If the second consumer of the message partition still consumes to the end displacement of the message partition, the message will be consumed repeatedly. This can avoid repeated consumption of messages.
[0073] For ease of understanding, the following Figure 2 An example is given to illustrate the consumption process of the first consumer and the second consumer of any partition. Figure 2 A schematic diagram of the consumption displacement of a message partition provided in an embodiment of the present application, the message partition includes offset1 to offset10, and the end displacement is offset10. It can be seen that when in the message accumulation state, this is the first stage, and the position method in the consumer monitoring container instance can be called to view that the first consumption displacement of the first consumer of the message partition is offset4. At this time, the first consumer can be controlled to stop consumption, and then the endOffsets method in the consumer monitoring container instance is called to obtain the end displacement of the message partition as offset10. Then, entering the second stage, the seek method of the consumer monitoring container instance can be called to specify the first consumer to start consuming from offset10. At the same time, create a second consumer, call the seek method of the consumer listening container to specify that the second consumer starts consuming from offset4. It should be noted that after consuming the message, the consumer needs to submit the consumption displacement so that the server can determine the current consumption displacement of the consumer. However, what is "submitted" is the next bit of the displacement actually consumed by the consumer, that is, when the consumer has consumed offsetx, what is submitted is offsetx+1. So the first consumption displacement of the above message partition is also the same. The message in the first consumption displacement of the message partition has not been consumed yet, so specifying the second consumer to consume from offset4 will not lead to repeated consumption, because the message actually consumed by the first consumer is the message in offset3, which will not be repeated here. Further, after detecting that the second consumer consumes to offset9, the topic subscribed by the second consumer can be cancelled to stop the second consumer from consuming, that is, the consumption area of the second consumer is offset4-offset9.
[0074] It can be seen that in this way, the first consumer of the message partition can directly start consuming the latest message, preventing message delays and greatly improving the quality of business processing. By constructing a new consumer and binding it to the second consumer group, and setting the consumption displacement of the second consumer of the message partition to the first consumption displacement of each partition through the seek method, the second consumer of the message partition can start consuming messages from the first consumption displacement of the message partition. In this way, the accumulated messages can be processed while ensuring that the latest messages will not be delayed, and messages can be prevented from being consumed repeatedly.
[0075] It should be noted that when the messages in the message partition are piled up, the first consumer can be controlled to stop consuming messages by calling the fourth method in the consumer listening container instance. The fourth method can be a stop method. After the consumption displacement of the first consumer of the message partition is adjusted to the end displacement of the message partition, the first consumer of the message partition can be controlled to start consuming by calling the fifth method of the consumer listening container instance. The fifth method can be a start method, which will not be repeated here.
[0076] In a possible embodiment, when the message consumption speed of the second consumer is less than the message consumption speed of the message partition configuration, n-1 second consumers may be newly created, where n is determined by the ratio of the message consumption speed of the message partition configuration to the message consumption speed of the second consumer. For example, if the message consumption speed of the message partition configuration is 3 and the message consumption speed of the second consumer is 1, then two second consumers may be newly created, and the three second consumers may simultaneously consume the accumulated messages in the message partition. Then, the area between the first consumption displacement and the end displacement is divided into n areas, and the starting consumption displacement of each area and the ending consumption displacement of each area are determined. Then, the n second consumers are controlled to consume the messages of the n areas at the same time, and each second consumer starts to consume messages from the starting consumption displacement of each area. Finally, the message displacement of the n second consumers is monitored. If the consumption displacement of each second consumer is the ending consumption displacement of each area, each second consumer is controlled to stop consuming messages.
[0077] Specifically, taking Kafka as an example, the pseudo code is as follows:
[0078] int offsetStart = offsetA;
[0079] int offsetEnd = 0;
[0080] for(int i=1;i<=n;i++){
[0081] / / Start segmentation
[0082] if(I == n){
[0083] offsetEnd = offsetB - 1;
[0084] }
[0085]
[0086] By analogy, it can be understood that each segmented message should be consumed simultaneously by the corresponding second consumer, so as to ensure that the consumption speed of the accumulated messages meets the message consumption speed configured by the message partition, thereby preventing business delays.
[0087] In combination with the above content, an example of a message processing method in an embodiment of the present application is introduced below.
[0088] See also Figure 3 , Figure 3 A flow chart of a message processing method provided in an embodiment of the present application specifically includes the following steps:
[0089] Step 301: When messages in a message partition are piled up, obtain a first consumer displacement of a first consumer of the message partition and an end displacement of the message partition.
[0090] The first consumption displacement is the position information of the message that the first consumer has consumed, and the first consumer is a consumer that exists in the current message partition itself.
[0091] Among them, when it is detected that the difference between the speed of adding new messages and the speed of consuming messages in the message partition is greater than the speed threshold, it can be determined that the message partition is in a message accumulation state. Specifically, the message processing method of the embodiment of the present application can be started through a message processing trigger service. The message processing trigger service can be a Nacos configuration center. Two configurations can be provided by the Nacos configuration center. The first configuration is the topic of message accumulation, and the second configuration is whether to trigger processing. The server can start a thread to loop and listen to the above configuration. When it is determined that message accumulation occurs and processing is triggered, the identifier of the topic can be read to determine the corresponding message partition, and the method of step 301 is executed.
[0092] Wherein, when messages in a message partition are piled up, the first consumer displacement can be obtained by calling a first method in a consumer monitoring container instance; and the end displacement can be obtained by calling a second method in the consumer monitoring container instance.
[0093] The creation of a consumer listening container instance can be performed by the springboot framework scanning the annotations of the consumer method. It can be understood that the consumer listening container instance can be used to consume messages. Specifically, the first method can be a position query function such as the position method, which is used to query the first consumption displacement of each message partition in the topic, and the second method can be an end position query function such as the endOffsets method, which is used to query the end displacement of each message partition in the topic. It can be understood that at this time, the first consumer of the message partition consumes the message to the first consumption displacement of the message partition. At this time, new messages are still being sent to the message queue middleware, and the first consumer of the message partition cannot consume the latest message in time.
[0094] In a possible embodiment, when messages in a message partition are accumulated, the first consumer may be controlled to stop consuming messages by calling a fourth method in the consumer listening container instance. The fourth method may be a stop method.
[0095] In this way, data support can be provided for the subsequent adjustment of the message consumption order.
[0096] Step 302: When the first consumption displacement is adjusted to the terminal displacement, the first consumer consumes messages from the terminal displacement.
[0097] The consumption displacement of the first consumer can be adjusted to the end displacement of each partition by calling the third method of the consumer monitoring container instance, so that the first consumer of the message partition starts consuming messages from the end displacement of the message partition. The third method can be a search function such as a seek method, which is used to specify that the first consumer of the message partition starts consuming from the end displacement of the message partition.
[0098] In a possible embodiment, the first consumer may be controlled to consume messages from the end displacement by calling a fifth method in the consumer monitoring container instance, and the fifth method may be a start method.
[0099] In this way, the first consumer of the message partition can directly start consuming the latest message, preventing message delays and greatly improving the quality of business processing.
[0100] Step 303: determine a second consumer of the message partition, and the second consumer starts consuming messages from the first consumption offset.
[0101] Among them, a second consumer group can be constructed by setting consumer parameters, and the second consumer group includes the second consumer; by calling the third method in the consumer monitoring container instance, the consumption displacement of the second consumer is adjusted to the first consumption displacement, so that the second consumer of the message partition starts to consume messages from the first consumption displacement of the message partition. It can be understood that the second consumer is a temporary consumer constructed to consume accumulated messages.
[0102] It can be seen that by constructing a new consumer and binding it to the second consumer group, and setting the consumption displacement of the second consumer of the message partition to the first consumption displacement of the message partition through the seek method, the second consumer of the message partition can start consuming messages from the first consumption displacement of the message partition. In this way, the accumulated messages can be processed while ensuring that the latest messages will not be delayed.
[0103] Step 304: monitor the message displacement of the second consumer. If the consumption displacement of the second consumer is one digit before the end displacement, the second consumer stops consuming messages.
[0104] The second consumer group may be controlled to stop consuming messages by unsubscribing from the topic.
[0105] It can be seen that if the second consumer of the message partition consumes to the position before the end displacement of the message partition, the subscription to the topic will be cancelled. This is because the first consumer of the message partition has started consuming from the end displacement of the message partition. If the second consumer of each partition still consumes to the end displacement of each partition, the message will be consumed repeatedly. This can avoid the repeated consumption of messages. It should be noted that after consuming the message, the consumer needs to submit the consumption displacement so that the server can determine the consumer's current consumption displacement. However, what is "submitted" is the next position of the displacement actually consumed by the consumer, that is, when the consumer has consumed offsetx, what is submitted is offsetx+1. Therefore, the first consumption displacement of the above message partition is also the same. The message in the first consumption displacement of the message partition has not been consumed yet.
[0106] It can be seen that through the above message processing method, first, when the messages in the message partition are piled up, the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition are obtained, and the first consumption displacement is the position information of the message that the first consumer has consumed; when the first consumption displacement is adjusted to the end displacement, the first consumer consumes messages from the end displacement; the second consumer of the message partition is determined, and the second consumer starts to consume messages from the first consumption displacement; the message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is the previous position of the end displacement, the second consumer is controlled to stop consuming messages. In the case of message accumulation, the accumulated messages and the newly generated messages can be isolated and processed in parallel, which greatly alleviates the message accumulation and can process the newly generated messages in time to avoid message delays.
[0107] For the parts not described in detail above, please refer to the description of the steps of the message processing method in the above specification, which will not be repeated here.
[0108] In combination with the above content, another message processing method of an embodiment of the present application is introduced as an example below.
[0109] See also Figure 4 , Figure 4 A flowchart of another message processing method provided in an embodiment of the present application specifically includes the following steps:
[0110] Step 401: When messages in a message partition are piled up, obtain a first consumer displacement of a first consumer of the message partition and an end displacement of the message partition.
[0111] Step 402: When the first consumption displacement is adjusted to the end displacement, the first consumer consumes messages from the end displacement.
[0112] Step 403: determine the second consumer of the message partition.
[0113] Step 404: When the message consumption speed of the second consumer is lower than the message consumption speed configured by the message partition, n-1 new second consumers are created.
[0114] Here, n is determined by the ratio of the message consumption speed of the message partition configuration to the message consumption speed of the second consumer.
[0115] Step 405: divide the area between the first consumption displacement and the terminal displacement into n sections, and determine the starting consumption displacement of each section and the ending consumption displacement of each section.
[0116] Step 406: Control n second consumers to consume messages from the n regions simultaneously, and each second consumer starts consuming messages from the start consumption displacement of each region.
[0117] Step 407: monitor the message displacements of the n second consumers. If the consumption displacement of each second consumer is the termination consumption displacement of each area, each second consumer stops consuming messages.
[0118] It can be seen that through the above message processing method, first, when the messages in the message partition are piled up, the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition are obtained, and the first consumption displacement is the position information of the message that the first consumer has consumed; when the first consumption displacement is adjusted to the end displacement, the first consumer consumes the message from the end displacement; the second consumer of the message partition is determined, and the second consumer starts to consume the message from the first consumption displacement; the message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is the previous one of the end displacement, the second consumer stops consuming the message. In the case of message accumulation, the accumulated messages and the newly generated messages can be isolated and processed in parallel, which greatly alleviates the message accumulation and can process the newly generated messages in time to avoid message delays. In addition, when the message consumption speed of the second consumer is less than the message consumption speed configured by the message partition, each segmented message should be consumed by the corresponding second consumer at the same time, so that the consumption speed of the accumulated messages can be guaranteed to meet the message consumption speed configured by the message partition, and business delays can be prevented.
[0119] For the parts not described in detail above, please refer to the description of the steps of the message processing method in the above specification, which will not be repeated here.
[0120] IV. Exemplary description of an electronic device
[0121] Combine the following Figure 5 An electronic device in an embodiment of the present application is described. Figure 5 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 includes a processor 501 , a memory 502 , and a communication bus 503 for connecting the processor 501 and the memory 502 .
[0122] In some possible implementations, the memory 502 includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or portable read-only memory (CD-ROM), and the memory 502 is used to store the program code executed by the electronic device 500 and the transmitted data.
[0123] In some possible implementations, the electronic device 500 further includes a communication interface for receiving and sending data.
[0124] In some possible implementations, the processor 501 may be one or more central processing units (CPUs). When the processor 501 is a central processing unit (CPU), the central processing unit (CPU) may be a single-core central processing unit (CPU) or a multi-core central processing unit (CPU).
[0125] In some possible implementations, the processor 501 may be a baseband chip, a chip, a central processing unit (CPU), a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof.
[0126] In a specific implementation, the processor 501 in the electronic device 500 is used to execute the program instructions 521 stored in the memory 502 to perform the following operations:
[0127] When messages in a message partition are piled up, obtaining a first consumption displacement of a first consumer of the message partition and an end displacement of the message partition, wherein the first consumption displacement is position information of messages that have been consumed by the first consumer;
[0128] In case the first consumption displacement is adjusted to the end displacement, the first consumer consumes messages from the end displacement;
[0129] Determine a second consumer of the message partition, and the second consumer starts consuming messages from the first consumption shift;
[0130] The message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is one digit before the end displacement, the second consumer stops consuming messages.
[0131] It can be seen that through the above-mentioned message processing method and related devices, first, when the messages in the message partition are piled up, the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition are obtained, and the first consumption displacement is the position information of the message that the first consumer has consumed; when the first consumption displacement is adjusted to the end displacement, the first consumer consumes the message from the end displacement; the second consumer of the message partition is determined, and the second consumer starts to consume the message from the first consumption displacement; the message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is the previous position of the end displacement, the second consumer is controlled to stop consuming the message. In the case of message accumulation, the accumulated messages and the newly generated messages can be isolated and processed in parallel, which greatly alleviates the message accumulation and can process the newly generated messages in time to avoid message delays.
[0132] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above, and the electronic device 500 can be used to execute the above method embodiment of the present application, which will not be described in detail.
[0133] V. Exemplary Description of a Message Processing Device
[0134] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0135] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0136] In the case of dividing each functional module into corresponding functional modules, Figure 6A functional unit composition block diagram of a message processing device provided in an embodiment of the present application, wherein the message processing device 600 includes:
[0137] An acquiring unit 610 is configured to acquire, when messages in a message partition are piled up, a first consumption displacement of a first consumer of the message partition and an end displacement of the message partition, wherein the first consumption displacement is position information of messages that have been consumed by the first consumer;
[0138] A first adjusting unit 620 is configured to, when the first consumption displacement is adjusted to the end displacement, cause the first consumer to consume messages from the end displacement;
[0139] A second adjustment unit 630 is used to determine a second consumer of the message partition, and the second consumer starts consuming messages from the first consumption displacement;
[0140] The consumption control unit 640 is used to monitor the message displacement of the second consumer. If the consumption displacement of the second consumer is one digit before the end displacement, the second consumer stops consuming messages.
[0141] In a possible embodiment, in terms of obtaining the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition, the obtaining unit 610 is specifically configured to:
[0142] When it is detected that the difference between the speed of adding new messages in the message partition and the speed of consuming messages is greater than a speed threshold, the first consumption displacement is obtained by calling a first method in the consumer monitoring container instance;
[0143] The end displacement is obtained by calling a second method in the consumer listening container instance.
[0144] In a possible embodiment, after obtaining the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition, the first adjustment unit 620 is further configured to:
[0145] By calling a third method in the consumer listening container instance, the consumption displacement of the first consumer is adjusted to the end displacement of each partition.
[0146] In a possible embodiment, in determining the second consumer of the message partition, and the second consumer starting to consume messages from the first consumption displacement, the second adjustment unit 630 is specifically used to:
[0147] Building a second consumer group by setting consumer parameters, wherein the second consumer group includes the second consumer;
[0148] By calling a third method in the consumer monitoring container instance, the consumption displacement of the second consumer is adjusted to the first consumption displacement.
[0149] In a possible embodiment, in terms of controlling the second consumer to stop consuming messages, the second adjusting unit is further configured to:
[0150] The second consumer group is unsubscribed from the topic, and the second consumer is controlled to stop consuming messages.
[0151] In a possible embodiment, the first adjusting unit 610 is further configured to:
[0152] When the messages in the message partition are accumulated, the first consumer is controlled to stop consuming messages by calling the fourth method in the consumer listening container instance.
[0153] In a possible embodiment, in terms of the first consumer consuming the message from the end position, the first adjusting unit 610 is specifically configured to:
[0154] The first consumer is controlled to consume messages from the end displacement by calling the fifth method in the consumer listening container instance.
[0155] In one possible embodiment,
[0156] The second adjustment unit is also used to create n-1 new second consumers when the message consumption speed of the second consumer is less than the message consumption speed of the message partition configuration, where n is determined by the ratio of the message consumption speed of the message partition configuration to the message consumption speed of the second consumer; divide the area between the first consumption displacement and the end displacement into n areas, determine the starting consumption displacement of each area and the ending consumption displacement of each area; control the n second consumers to consume messages from the n areas at the same time, and each second consumer starts to consume messages from the starting consumption displacement of each area; monitor the message displacement of the n second consumers, and if the consumption displacement of each second consumer is the ending consumption displacement of each area, each second consumer stops consuming messages.
[0157] It can be seen that through the above-mentioned message processing method and related devices, first, when the messages in the message partition are piled up, the first consumption displacement of the first consumer of the message partition and the end displacement of the message partition are obtained, and the first consumption displacement is the position information of the message that the first consumer has consumed; when the first consumption displacement is adjusted to the end displacement, the first consumer consumes the message from the end displacement; the second consumer of the message partition is determined, and the second consumer starts to consume the message from the first consumption displacement; the message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is the previous one of the end displacement, the second consumer is controlled to stop consuming the message. In the case of message accumulation, the accumulated messages and the newly generated messages can be isolated and processed in parallel, which greatly alleviates the message accumulation and can process the newly generated messages in time to avoid message delays. In addition, when the message consumption speed of the second consumer is less than the message consumption speed configured by the message partition, each segmented message should be consumed by the corresponding second consumer at the same time, so that the consumption speed of the accumulated messages can be guaranteed to meet the message consumption speed configured by the message partition, and business delays can be prevented.
[0158] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above, and the message processing device 600 can be used to execute the above method embodiment of the present application, which will not be repeated here.
[0159] 6. Other examples
[0160] An embodiment of the present application also provides a chip, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiment.
[0161] An embodiment of the present application also provides a chip module, including a transceiver component and a chip, the chip including a processor, a memory and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiment.
[0162] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiment, and the above computer includes an electronic device.
[0163] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.
[0164] It should be noted that, for the above-mentioned various embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions, steps, modules or units involved are not necessarily required by the embodiments of the present application.
[0165] In the above embodiments, the embodiments of the present application have different focuses on the description of each embodiment. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0166] The steps of the method or algorithm described in the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, read-only compact disks (CD-ROMs) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device or a management device. Of course, the processor and the storage medium can also be present in a terminal device or a management device as discrete components.
[0167] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).
[0168] The modules / units included in the devices and products described in the above embodiments may be software modules / units or hardware modules / units, or may be partially software modules / units and partially hardware modules / units. For example, for the devices and products applied to or integrated in the chip, the modules / units included therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, which run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits; for the devices and products applied to or integrated in the chip module, the modules / units included therein may all be implemented in the form of hardware such as circuits, and different modules / units may be located in the same component (such as a chip, circuit module, etc.) or in different components of the chip module, or at least some of the modules / units may be implemented in the form of software programs. The software programs run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.
[0169] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above description is only the specific implementation method of the embodiments of the present application and is not intended to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.
Claims
1. A message processing method, characterized in that: The method comprises: When messages in a message partition are piled up, obtaining a first consumption displacement of a first consumer of the message partition and an end displacement of the message partition, wherein the first consumption displacement is position information of messages that have been consumed by the first consumer; In case the first consumption displacement is adjusted to the end displacement, the first consumer consumes messages from the end displacement; Determine a second consumer of the message partition, and the second consumer starts consuming messages from the first consumption displacement; The message displacement of the second consumer is monitored, and if the consumption displacement of the second consumer is one digit before the end displacement, the second consumer stops consuming messages.
2. The method according to claim 1, characterized in that The obtaining the first consumer displacement of the first consumer of the message partition and the end displacement of the message partition includes: When it is detected that the difference between the speed of adding new messages in the message partition and the speed of consuming messages is greater than a speed threshold, the first consumption displacement is obtained by calling a first method in the consumer monitoring container instance; The end displacement is obtained by calling a second method in the consumer listening container instance.
3. The method according to claim 1, characterized in that After obtaining the first consumer displacement of the first consumer of the message partition and the end displacement of the message partition, the method further includes: By calling the third method in the consumer listening container instance, the consumption displacement of the first consumer is adjusted to the end displacement of the message partition.
4. The method according to claim 1, characterized in that: The determining of a second consumer of the message partition, and the second consumer consuming messages starting from the first consumption shift, comprises: Building a second consumer group by setting consumer parameters, wherein the second consumer group includes the second consumer; By calling a third method in the consumer listening container instance, the consumption displacement of the second consumer is adjusted to the first consumption displacement.
5. The method according to claim 1, characterized in that: The method further comprises: When the messages in the message partition are accumulated, the first consumer is controlled to stop consuming messages by calling the fourth method in the consumer listening container instance.
6. The method according to claim 5, characterized in that The first consumer consumes the message from the end position, including: The first consumer is controlled to consume messages from the end displacement by calling the fifth method in the consumer listening container instance.
7. The method according to claim 1, characterized in that The method further comprises: When the message consumption speed of the second consumer is lower than the message consumption speed of the message partition configuration, n-1 second consumers are newly created, where n is determined by the ratio of the message consumption speed of the message partition configuration to the message consumption speed of the second consumer; Divide the area between the first consumption displacement and the terminal displacement into n sections, and determine the starting consumption displacement of each section and the ending consumption displacement of each section; Controlling n second consumers to consume messages from the n regions simultaneously, each second consumer starting to consume messages from a starting consumption displacement of each region; The message displacements of the n second consumers are monitored, and if the consumption displacement of each second consumer is the termination consumption displacement of each area, each second consumer is controlled to stop consuming messages.
8. A message processing device, characterized in that: The device comprises: an acquiring unit, configured to acquire, when messages in a message partition are piled up, a first consumption displacement of a first consumer of the message partition and an end displacement of the message partition, wherein the first consumption displacement is position information of messages that have been consumed by the first consumer; A first adjusting unit, configured to cause the first consumer to consume messages from the terminal displacement when the first consumption displacement is adjusted to the terminal displacement; A second adjustment unit is used to determine a second consumer of the message partition, and the second consumer starts consuming messages from the first consumption displacement; The consumption control unit is used to monitor the message displacement of the second consumer. If the consumption displacement of the second consumer is one digit before the end displacement, the second consumer stops consuming messages.
9. An electronic device, characterized in that: include: A processor, a memory, and one or more programs; the one or more programs are stored in the memory and are configured to be executed by the processor, the programs comprising instructions for executing the steps in the method according to any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.