Message processing method, device, electronic device and message processing system

Abstract

The present application proposes a message processing method, device, electronic device and message processing system, which relates to the field of computer technology, wherein the method is applied to a message processing node, and the method includes: obtaining the processing status information of the message in the first message topic processed by the message processing node; when it is determined that the processing status information meets the first preset condition, forwarding the message in the first message topic to multiple second message topics; and processing the messages in multiple second message topics in parallel. Thus, the machine configuration resources can be efficiently utilized when the number of message processing nodes is fixed, so that the machine cost consumption is low and the utilization rate of the machine configuration resources is high. Moreover, since the message processing node can process the messages in multiple second message topics in parallel, it is possible to realize rapid processing of messages and ensure the timeliness of messages.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a message processing method, device, electronic device and message processing system. Background Art

[0002] With the development of Internet technology and the growth of distributed systems, the application of distributed messaging systems is becoming more and more widespread. Currently, the more common distributed messaging systems include Kafka, Rocketmq, etc. A messaging system usually includes a Producer (message generator, also known as a producer), a server cluster consisting of one or more Brokers (servers), and a Consumer (message processor, also known as a consumer). The message generator is responsible for generating and publishing messages to the server cluster, and the message processor is responsible for reading messages from the server cluster and executing corresponding business logic processing.

[0003] Related technologies usually increase the number of machines on the message processing side to increase the message processing speed in the server cluster to reduce the backlog of unprocessed messages. This method consumes a lot of machine costs on the message processing side, and the configuration resources such as memory and CPU (Central Processing Unit) of each machine cannot be fully utilized, and the utilization rate of machine configuration resources is low. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] The embodiments of the present application propose a message processing method, device, electronic device and message processing system to solve the technical problems of high machine cost consumption and low utilization rate of machine configuration resources existing in the message processing method in the related art.

[0006] The first aspect of the present application proposes a message processing method, which is applied to a message processing node, including: obtaining processing status information of a message in a first message topic processed by the message processing node; when it is determined that the processing status information satisfies a first preset condition, forwarding the message in the first message topic to multiple second message topics; and processing the messages in multiple second message topics in parallel.

[0007] The second aspect of the present application proposes a message processing device, which is applied to a message processing node, including: an acquisition module, used to obtain processing status information of a message in a first message topic processed by the message processing node; a forwarding module, used to forward the message in the first message topic to multiple second message topics when it is determined that the processing status information meets a first preset condition; a first processing module, used to process messages in multiple second message topics in parallel.

[0008] The third aspect embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the message processing method proposed in the first aspect embodiment of the present application.

[0009] The fourth aspect of the present application proposes a message processing system, including: a message generator, a server cluster, and a message processor, wherein at least one message topic is distributed in the server cluster; wherein the message generator is used to generate a message, the message has a message topic to which it belongs, and is used to publish the message to the corresponding message topic in the server cluster; the server cluster is used to receive the message published by the message generator and store the message in the corresponding message topic; the message processor includes at least one message processing node, and the message processing node is used to execute the method described in the first aspect of the present application to process the message in the first message topic of the at least one message topic.

[0010] The fifth aspect embodiment of the present application proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the message processing method proposed in the first aspect embodiment of the present application.

[0011] The sixth aspect embodiment of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the message processing method proposed in the first aspect embodiment of the present application.

[0012] One embodiment of the above invention has the following advantages or beneficial effects:

[0013] The processing status information of the message in the first message topic processed by the message processing node is obtained through the message processing node. When it is determined that the processing status information satisfies the first preset condition, the message in the first message topic is forwarded to multiple second message topics, and the messages in the multiple second message topics are processed in parallel. In this way, when the number of message processing nodes is fixed, the machine configuration resources can be efficiently utilized, so that the machine cost consumption is low and the utilization rate of the machine configuration resources is high. Moreover, since the message processing node can process the messages in multiple second message topics in parallel, the messages can be quickly processed to ensure the timeliness of the messages.

[0014] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0016] Figure 1 A schematic diagram of the structure of the message processing system provided in Example 1 of the present application;

[0017] Figure 2 A flowchart of a message processing method provided in Embodiment 2 of the present application;

[0018] Figure 3 An example diagram of a message processing node provided in Embodiment 2 of the present application forwarding and processing a message in message topic a;

[0019] Figure 4 A flowchart of a message processing method provided in Embodiment 3 of the present application;

[0020] Figure 5 This is an example diagram of a message processing node forwarding a message provided in Embodiment 3 of the present application;

[0021] Figure 6 A flowchart of a message processing method provided in Embodiment 4 of the present application;

[0022] Figure 7 This is a scenario example diagram of the message processing method provided in Embodiment 4 of the present application;

[0023] Figure 8 A schematic diagram of the structure of a message processing device provided in Embodiment 5 of the present application;

[0024] Fig. 9 A schematic diagram of the structure of a message processing device provided in Embodiment 6 of the present application;

[0025] Fig.10 A block diagram of an exemplary electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] The embodiments of the present application aim at the technical problems of large machine cost consumption and low utilization rate of machine configuration resources existing in the message processing methods in the related technologies, and propose a message processing method, device, electronic device, message processing system, storage medium and computer program product, wherein the method is applied to a message processing node, and the method includes: obtaining processing status information of a message in a first message topic processed by the message processing node; when it is determined that the processing status information satisfies a first preset condition, forwarding the message in the first message topic to multiple second message topics; and processing the messages in multiple second message topics in parallel. Thus, the machine configuration resources can be efficiently utilized when the number of message processing nodes is fixed, so that the machine cost consumption is low and the utilization rate of machine configuration resources is high, and because the message processing node can process the messages in multiple second message topics in parallel, it is possible to realize rapid processing of messages and ensure the timeliness of messages.

[0029] The following describes the message processing method, device, electronic device, message processing system, storage medium and computer program product of the embodiments of the present application with reference to the accompanying drawings.

[0030] The following first describes the message processing system provided in the embodiment of the present application.

[0031] Figure 1 A structural diagram of the message processing system provided in Example 1 of the present application.

[0032] like Figure 1 As shown, the message processing system may include a message generator 110, a server cluster 120, and a message processor 130. The server cluster 120 is composed of a plurality of servers, and at least one message topic is distributed in the server cluster.

[0033] The message generator 110 is used to generate a message. The generated message has a message topic. The message generator 110 is also used to publish the message to the corresponding message topic in the server cluster.

[0034] The server cluster 120 is used to receive the message published by the message generator 110 and store the message in the corresponding message topic;

[0035] The message processor 130 includes at least one message processing node, which is used to execute the message processing method of the embodiment of the present application to process the message in the first message topic in at least one message topic.

[0036] in, Figure 1 In the figure, it is taken as an example that the server cluster 120 is distributed with message topic a and message topic b, and the message processor 130 includes message processing node A, message processing node B, message processing node C, and message processing node D.

[0037] Among them, the message topic is the logical channel for sending and receiving messages. A message topic can physically include multiple partitions, and each partition can be composed of a series of orderly arranged and immutable messages, that is, each partition can be regarded as an ordered message queue. Since the types of messages in each partition of a message topic are the same, in the embodiment of the present application, the messages in a message topic can be regarded as a message queue.

[0038] The first message topic may be any message topic distributed in the server cluster.

[0039] The message processing node can be an independently configured processor of the message processor 130, or a cluster composed of multiple independently configured processors of the message processor 130, or a device configured in an independently configured processor of the message processor 130. This application does not impose any restrictions on this.

[0040] refer to Figure 1 , the message generated by the message generator 110 may belong to message topic a and message topic b. The message generator 110 may publish the message belonging to message topic a to the corresponding message topic a in the server cluster 120, and publish the message belonging to message topic b to the corresponding message topic b in the server cluster 120. Correspondingly, the server cluster 120 may receive the message published by the message generator 110, and store the message belonging to message topic a in the corresponding message topic a, and store the message belonging to message topic b in the corresponding message topic b.

[0041] The message processing nodes A, B, and C in the message processing party 130 can respectively execute the message processing method of the embodiment of the present application to process the message in the message subject a. The message processing node D in the message processing party 130 can execute the message processing method of the embodiment of the present application to process the message in the message subject b. Processing the message means executing the corresponding business processing logic according to the content contained in the message.

[0042] The message processing method provided in the embodiment of the present application is described below.

[0043] Figure 2 This is a flow chart of the message processing method provided in Example 2 of this application. Figure 2 As shown, the message processing method may include the following steps 201-203.

[0044] Step 201: Obtain processing status information of a message in a first message topic processed by a message processing node.

[0045] It should be noted that the message processing method provided in the embodiment of the present application is applied to a message processing node and can be executed by a message processing device. The message processing device can be a message processing node, or can be configured in a message processing node, so as to reduce machine cost consumption, improve the utilization rate of machine configuration resources, and realize rapid message processing to ensure the timeliness of messages by executing the method. The embodiment of the present application is described by taking the message processing device as a message processing node as an example.

[0046] The message processing node may be any message processing node in the message processing party. The first message topic is the message topic to which the message processed by the message processing node belongs, and the first message topic may be any message topic distributed in the server cluster.

[0047] For example, the first message subject can be Figure 1 The message topic a shown in the figure, correspondingly, the message processing node can be Figure 1 The message processing node A, message processing node B or message processing node C shown. Alternatively, the first message topic can be Figure 1 The message subject b shown in the figure, correspondingly, the message processing node can be Figure 1 The message processing node D shown is shown. For each message processing node, the message in the first message topic can be processed according to the above message processing method.

[0048] The processing status information may include any information that can reflect the processing status of the messages in the first message topic, such as the message backlog and the message processing rate of the messages in the first message topic. Among them, the message backlog indicates the total number of unprocessed messages in the first message topic. The message processing rate indicates the speed at which the messages in the first message topic are processed.

[0049] Step 202: when it is determined that the processing status information satisfies a first preset condition, forward the message in the first message topic to multiple second message topics.

[0050] Step 203: Process messages in multiple second message topics in parallel.

[0051] In an embodiment of the present application, the message processing node can be set to process the message in the first message topic in a processing mode including a single-topic processing mode and a multi-topic processing mode. In the single-topic processing mode, the message processing node directly processes the message in the first message topic; in the multi-topic processing mode, the message processing node forwards the message in the first message topic to multiple second message topics, and then processes the messages in multiple second message topics in parallel.

[0052] Among them, the first preset condition is the condition that needs to be met for processing status information when switching from a single-topic processing mode to a multi-topic processing mode. The first preset condition can be set as needed. For example, the first preset condition can be set to a message backlog greater than a first preset threshold. Alternatively, the first preset condition can be set to a message processing rate lower than a third preset threshold. Among them, the first preset threshold and the third preset threshold can be set arbitrarily as needed, and this application does not limit this.

[0053] The number of second message topics can be determined as needed. For example, it can be determined based on the number of message processing nodes included in the message processing party, the machine configuration resources of the message processing nodes, and / or the rate of increase of messages in the first message topic. For example, when the number of message processing nodes included in the message processing party is small, and the machine configuration resources of the message processing nodes are low, in order to avoid the excessive number of messages processed in parallel by each message processing node, resulting in the inability of the machine configuration resources to bear, the number of second message topics can be set to be small; when the number of message processing nodes included in the message processing party is large, and the machine configuration resources of the message processing nodes are high, in order to make full use of the machine configuration resources of the message processing nodes, the number of second message topics can be set to be large; when the number of message processing nodes included in the message processing party is small, and the rate of increase of messages in the first message topic is fast, in order to avoid the excessive number of messages processed in parallel by each message processing node, resulting in the inability of the machine configuration resources to bear, the number of second message topics can be set to be small, etc.

[0054] In an example embodiment, the message processing node may process the message in the first message topic in a single-topic processing mode as a regular processing mode, and monitor the processing status information of the message in the first message topic processed by it in real time or at a preset frequency. When the message processing node determines that the processing status information satisfies the first preset condition, the message in the first message topic may be forwarded to multiple second message topics, and then the message in the first message topic may be processed in a multi-topic processing mode, that is, the corresponding business logic processing is performed on the messages in multiple second message topics in parallel.

[0055] For example, refer to Figure 3 , assuming that the number of second message topics is 3, including message topic 1, message topic 2 and message topic 3. Message processing node A, message processing node B and message processing node C can all use the single topic processing mode as the regular processing mode to process the messages in the first message topic, i.e., message topic a, and monitor the processing status information of the messages in the message topic a they process in real time or at a preset frequency. When message processing node A, message processing node B and message processing node C determine that the processing status information of the messages in message topic a meets the first preset condition, the messages in message topic a can be forwarded to the three second message topics, i.e., Figure 3 , and then process the messages in the three second message topics in parallel. Therefore, in the single-topic processing mode, each message processing node can process one message in the first message topic at the same time, and when the processing status information of the message meets the first preset condition, each message processing node can process the messages in the three second message topics, i.e., three messages, in parallel at the same time, so that the processing status information of the message in the first message topic meets the first preset condition, such as when the message backlog is greater than the first preset threshold, when the number of message processing nodes in the message processing party is fixed, the machine configuration resources of each message processing node can be efficiently utilized, the machine cost consumption is low, the utilization rate of the machine configuration resources is high, and because each message processing node can process the messages in the three second message topics in parallel, it is possible to process the messages quickly and ensure the timeliness of the messages.

[0056] It should be noted that the drawings in the embodiments of the present application are illustrated by taking the case where each message topic has only one shard and is distributed on a server in a server cluster as an example. In actual applications, there may be situations where each message topic has multiple shards, or is distributed on multiple servers. Regardless of where the messages in each message topic are distributed, as long as the messages in the message topic can be obtained, the message processing method provided in the embodiments of the present application can be used to process the messages in the message topic. Therefore, the message processing method provided in the embodiments of the present application is applicable to any of the above situations.

[0057] In summary, the message processing method provided in the embodiment of the present application is applied to a message processing node. By obtaining the processing status information of the message in the first message topic processed by the message processing node, when it is determined that the processing status information satisfies the first preset condition, the message in the first message topic is forwarded to multiple second message topics, and the messages in the multiple second message topics are processed in parallel. In this way, when the number of message processing nodes is fixed, the machine configuration resources can be efficiently utilized, thereby reducing the machine cost consumption and increasing the utilization rate of the machine configuration resources. Moreover, since the message processing node can process the messages in multiple second message topics in parallel, the messages can be quickly processed to ensure the timeliness of the messages.

[0058] Combine the following Figure 4 , the message processing method in the embodiment of the present application is further explained.

[0059] Figure 4 This is a flow chart of the message processing method provided in Example 3 of this application. Figure 4 As shown, the message processing method may include the following steps 401-404.

[0060] Step 401: Acquire processing status information of messages in a first message topic processed by a message processing node, wherein the processing status information includes a message backlog.

[0061] The message backlog represents the total number of unprocessed messages in the first message topic.

[0062] Step 402: when it is determined that the message backlog is greater than a first preset threshold, forward the message in the first message topic to a plurality of second message topics.

[0063] Among them, the first preset threshold can be set arbitrarily as needed, and this application does not impose any restrictions on this.

[0064] In an embodiment of the present application, when the message processing node determines that the message backlog is greater than a first preset threshold, the message processing node may forward the messages in the first message topic to multiple second message topics in the following manner: obtain a preset number of messages from the first message topic at a preset time interval; and randomly forward the preset number of messages obtained each time to any one of the multiple second message topics.

[0065] Among them, the preset time interval and the preset number can be set arbitrarily as needed, and this application does not impose any restrictions on this.

[0066] In an embodiment of the present application, it can be pre-specified to which second message topics the messages in the first message topic will be forwarded, and then the message processing node can obtain a preset number of messages from the first message topic at a preset time interval, and each time the obtained preset number of messages are forwarded, the obtained preset number of messages can be randomly forwarded to any one of the multiple second message topics.

[0067] As a possible implementation method, a corresponding serial number can be set in advance for each second message topic. For example, when the number of second message topics is M, the serial number corresponding to each second message topic can be set to 0 to M-1, and then the message processing node can generate a random number that is less than M and is an integer before forwarding the preset number of messages obtained each time, and then forward the obtained preset number of messages to the second message topic whose corresponding serial number is the generated random number.

[0068] refer to Figure 5 , assuming that the second message topics corresponding to the first message topic include message topic 1, message topic 2, and message topic 3, a corresponding serial number can be set in advance for each second message topic, for example, the serial number corresponding to message topic 1 is 0, the serial number corresponding to message topic 2 is 1, and the serial number corresponding to message topic 3 is 2. The message processing node A can obtain a preset number of messages from the first message topic at a preset time interval, and before forwarding the obtained preset number of messages to message topic 1, message topic 2, or message topic 3 each time, generate a random number that is less than 3 and is an integer, and then when the generated random number is 0, forward the obtained preset number of messages to message topic 1 corresponding to random number 0, when the generated random number is 1, forward the obtained preset number of messages to message topic 2 corresponding to random number 1, and when the generated random number is 2, forward the obtained preset number of messages to message topic 3 corresponding to random number 2.

[0069] Since random numbers naturally have a distributed characteristic, a message processing node obtains a preset number of messages from a first message topic at a preset time interval, and uses a generated random number to randomly forward the preset number of messages obtained each time to any one of a plurality of second message topics, so that the message processing node does not need to use a complex algorithm to determine how to forward the obtained messages to a plurality of second message topics, thereby achieving efficient forwarding of messages.

[0070] Step 403: Pull messages from the corresponding second message topics in parallel through the message pulling threads corresponding to the second message topics.

[0071] Step 404: Process the messages pulled from the second message topics in parallel.

[0072] In an embodiment of the present application, a message processing node and a message pulling thread corresponding to each second message topic can be pre-registered, so that after the message processing node forwards the message in the first message topic to multiple second message topics, the message pulling thread corresponding to each second message topic can be executed cyclically, and messages can be continuously pulled from the corresponding second message topics in parallel, and then the messages pulled from each second message topic can be processed in parallel. In this way, the message processing node can pull and process messages from each second message topic in parallel, so that the message can be processed quickly and the timeliness of the message can be guaranteed.

[0073] In addition, after the message processing node successfully pulls messages from each second message topic, it can also return a receipt message, such as ACK (Acknowledge character), to each second message topic of the server cluster, so that each second message topic of the server cluster confirms that the message processing node has successfully obtained the message.

[0074] In summary, the message processing method provided in the embodiment of the present application obtains the processing status information of the message in the first message topic processed by the message processing node through the message processing node, the processing status information includes the message backlog, and when it is determined that the message backlog is greater than the first preset threshold, the message in the first message topic is forwarded to multiple second message topics, and the message pulling threads corresponding to each second message topic are used to pull messages from the corresponding second message topics in parallel, and the messages pulled from each second message topic are processed in parallel. When the message backlog in the first message topic is high and the number of message processing nodes is fixed, the machine configuration resources can be efficiently utilized, thereby realizing rapid processing of the backlog of messages in the first message topic, ensuring the timeliness of the messages, and there is no need to increase the message processing nodes, the machine cost consumption is low, and the utilization rate of the machine configuration resources is high.

[0075] Through the above analysis, it can be known that when the processing status information of the message in the first message topic meets the first preset condition, the message processing node can process the message in the first message topic in a multi-topic processing mode. In a possible implementation form, it can also be restored to a single-topic processing mode to process the message in the first message topic as needed. Figure 6 , the message processing method provided in the embodiment of the present application is further explained.

[0076] Figure 6 This is a flow chart of the message processing method provided in the fourth embodiment of the present application. Figure 6 As shown, the message processing method includes the following steps 601-604.

[0077] Step 601: Obtain processing status information of a message in a first message topic processed by a message processing node.

[0078] The processing status information may include the message backlog and the message forwarding volume within a preset time period. The message backlog indicates the total number of unprocessed messages in the first message topic. The message forwarding volume within a preset time period indicates the total number of messages in the first message topic forwarded to the second message topic within the preset time period. The preset time period may be set as needed, such as 1 second, 2 seconds, etc., and this application does not limit this.

[0079] Step 602: When it is determined that the processing status information satisfies a first preset condition, forward the message in the first message topic to multiple second message topics.

[0080] Among them, the first preset condition is the condition that the processing status information needs to meet when switching from the single-topic processing mode to the multi-topic processing mode.

[0081] The first preset condition can be set as needed, and this application does not limit this. For example, the first preset condition can be set to: the message backlog is greater than the first preset threshold;

[0082] Step 603: Process messages in multiple second message topics in parallel.

[0083] The specific implementation process and principle of steps 601-603 may refer to the description of the above embodiment and will not be repeated here.

[0084] Step 604: When it is determined that the processing status information satisfies the second preset condition, stop forwarding the messages in the first message topic to the plurality of second message topics, and process the messages in the first message topic.

[0085] Among them, the second preset condition is the condition that the processing status information needs to meet when switching from the multi-topic processing mode to the single-topic processing mode.

[0086] The second preset condition can be set as needed, and this application does not limit this. For example, the second preset condition can be set to: the message backlog is not greater than the first preset threshold, and the message forwarding volume within the preset time period is less than the second preset threshold. Among them, the second preset threshold can be set arbitrarily as needed, and this application does not limit this.

[0087] In an embodiment of the present application, by stopping forwarding the message in the first message topic to multiple second message topics and processing the message in the first message topic when it is determined that the processing status information satisfies the second preset condition, it can be achieved that when the processing status information of the message in the first message topic satisfies the second preset condition, each message processing node only processes one message in the first message topic at the same time, thereby reducing the machine configuration resource consumption of the message processing node and achieving rational use of machine configuration resources.

[0088] When the second preset condition is set to that the message backlog is not greater than the first preset threshold, and the message forwarding volume within the preset time period is less than the second preset threshold, correspondingly, step 604 may include: when it is determined that the message backlog is not greater than the first preset threshold, and the message forwarding volume within the preset time period is less than the second preset threshold, stopping forwarding the messages in the first message topic to multiple second message topics.

[0089] Since the message backlog is not greater than the first preset threshold and the message forwarding volume within the preset time period is less than the second preset threshold, there is no backlog risk for messages in the first message topic. Therefore, by stopping forwarding the messages in the first message topic to multiple second message topics in this case and processing the messages in the first message topic, it can be achieved that when there is no backlog risk for messages in the first message topic, each message processing node only processes one message in the first message topic at the same time, thereby reducing the machine configuration resource consumption of the message processing nodes and achieving rational use of machine configuration resources.

[0090] Reference below Figure 7 , taking the first preset condition that the message backlog J is greater than the first preset threshold Q1, the second preset condition that the message backlog J is not greater than the first preset threshold Q1, and the message forwarding volume Z within the preset time period is less than the second preset threshold Q2, and the multiple second message topics include message topic 1, message topic 2, and message topic 3 as an example, the message processing method provided in the embodiment of the present application is further described. Among them, Figure 7 The message processor in the first message topic includes at least one message processing node for processing the message in the first message topic. Figure 7 Only the message processor is used to represent each message processing node that processes the message in the first message topic.

[0091] refer to Figure 7 Under normal circumstances (the message backlog J in the first message topic a is not greater than the first preset threshold value Q1), the messages in the first message topic a can be processed quickly and efficiently by the message processing party, and the messages in the first message topic a can be allowed to increase instantly. At this time, each message processing node of the message processing party that processes the messages in the first message topic a can use the single-topic processing mode as the regular processing mode to process the messages in the first message topic a, and monitor the processing status information of the messages in the first message topic a processed by it in real time or at a preset frequency. When each message processing node uses the single-topic processing mode as the regular processing mode to process the messages in the first message topic a, each message processing node can process one message in the first message topic a at the same time.

[0092] When the speed at which the message generator generates messages continues to increase, the message processing speed cannot match the speed at which the messages are generated, resulting in a continuous backlog of messages. When each message processing node of the message processor processes the messages in the first message topic a and determines that the message backlog J in the first message topic a is greater than the first preset threshold value Q1, the multi-topic processing mode can be turned on, and the message forwarding is first performed, that is, the messages in the first message topic a are randomly forwarded to message topic 1, message topic 2, and message topic 3, and then each message processing node changes the processing mode of the messages in the first message topic a to the multi-topic processing mode, that is, the corresponding business logic processing is performed on the messages in message topic 1, message topic 2, and message topic 3 in parallel. When each message processing node processes messages in the multi-topic processing mode, each message processing node can process the messages in the three second message topics, that is, three messages, in parallel at the same time, so that the message processing can be performed quickly and efficiently if the machine configuration resources of the message processing node allow.

[0093] When each message processing node of the message processing party processes the message in the first message topic a and determines that the message backlog J in the first message topic a is not greater than the first preset threshold Q1, and the message forwarding volume Z within the preset time period is less than the second preset threshold Q2, it can switch to the single topic processing mode to process the message in the first message topic a, thereby reducing the machine configuration resource consumption of the message processing node under normal circumstances and realizing the rational use of machine configuration resources.

[0094] In summary, the message processing method provided in the embodiment of the present application obtains the processing status information of the message in the first message topic processed by the message processing node through the message processing node, and when it is determined that the processing status information satisfies the first preset condition, forwards the message in the first message topic to multiple second message topics, and processes the messages in the multiple second message topics in parallel, and when it is determined that the processing status information satisfies the second preset condition, stops forwarding the message in the first message topic to multiple second message topics, and processes the message in the first message topic. This can realize that when the number of message processing nodes is fixed, according to the processing status information of the message in the first message topic, the processing mode of the message processing node for the message in the first message topic can be dynamically adjusted, thereby realizing rapid processing of messages, ensuring the timeliness of messages, maximizing the use of machine configuration resources, and improving the utilization rate of machine configuration resources. The implementation method is simple and easy to implement.

[0095] Figure 8 This is a structural diagram of the message processing device provided in Example 5 of the present application.

[0096] like Figure 8 As shown, the message processing device 800 may include: an acquisition module 810 , a forwarding module 820 and a first processing module 830 .

[0097] The acquisition module 810 is used to acquire the processing status information of the message in the first message topic processed by the message processing node;

[0098] A forwarding module 820, configured to forward the message in the first message topic to multiple second message topics when determining that the processing status information satisfies a first preset condition;

[0099] The first processing module 830 is used to process messages in multiple second message topics in parallel.

[0100] It should be noted that the message processing device provided in the embodiment of the present application can execute the message processing method of the aforementioned embodiment. The message processing device can be a message processing node, and can also be configured in a message processing node. By executing the above method, it can reduce machine cost consumption, improve the utilization rate of machine configuration resources, and realize rapid processing of messages to ensure the timeliness of messages.

[0101] It should be noted that the explanations in the aforementioned message processing method embodiment are also applicable to the message processing device of this embodiment, and will not be repeated here.

[0102] The message processing device of the embodiment of the present application obtains the processing status information of the message in the first message topic processed by the message processing node, and when it is determined that the processing status information satisfies the first preset condition, forwards the message in the first message topic to multiple second message topics, and processes the messages in the multiple second message topics in parallel. This can efficiently utilize machine configuration resources when the number of message processing nodes is fixed, thereby reducing machine cost consumption and increasing the utilization rate of machine configuration resources. Moreover, since the message processing node can process the messages in multiple second message topics in parallel, it is possible to quickly process messages and ensure the timeliness of the messages.

[0103] Fig. 9 This is a structural diagram of the message processing device provided in Example 6 of the present application.

[0104] like Fig. 9 As shown, the message processing device 900 may include: an acquisition module 910, a forwarding module 920 and a first processing module 930. Fig. 9 The acquisition module 910, the forwarding module 920 and the first processing module 930, and Figure 8 The acquisition module 810, the forwarding module 820 and the first processing module 830 have the same function and structure.

[0105] It should be noted that the message processing device 900 provided in the embodiment of the present application can execute the message processing method of the aforementioned embodiment. The message processing device can be a message processing node, and can also be configured in a message processing node. By executing the above method, it can reduce machine cost consumption, improve the utilization rate of machine configuration resources, and realize rapid processing of messages to ensure the timeliness of messages.

[0106] In a possible implementation of the embodiment of the present application, the processing state information includes a message backlog;

[0107] The forwarding module 920 includes:

[0108] The forwarding unit is used to forward the messages in the first message topic to multiple second message topics when it is determined that the message backlog is greater than a first preset threshold.

[0109] In another possible implementation manner of the embodiment of the present application, the forwarding unit includes:

[0110] An acquisition subunit, used to acquire a preset number of messages from the first message topic at a preset time interval;

[0111] The forwarding subunit is used to randomly forward a preset number of messages obtained each time to any one of the multiple second message topics.

[0112] In another possible implementation of the embodiment of the present application, the first processing module 930 includes:

[0113] A pulling unit, configured to pull messages from the corresponding second message topics in parallel through the message pulling threads corresponding to the second message topics;

[0114] The first processing unit is used to process the messages pulled from the second message topics in parallel.

[0115] In another possible implementation of the embodiment of the present application, the message processing device 900 further includes:

[0116] The second processing module 940 is used to control the forwarding module to stop forwarding the messages in the first message topic to multiple second message topics and process the messages in the first message topic when it is determined that the processing status information meets the second preset condition.

[0117] In another possible implementation of the embodiment of the present application, the processing status information includes a message backlog and a message forwarding volume within a preset time period;

[0118] The second processing module 940 includes:

[0119] The second processing unit is used to control the forwarding module to stop forwarding messages in the first message topic to multiple second message topics when it is determined that the message backlog is not greater than the first preset threshold and the message forwarding volume within the preset time period is less than the second preset threshold.

[0120] The message processing device of the embodiment of the present application obtains the processing status information of the message in the first message topic processed by the message processing node, and when it is determined that the processing status information satisfies the first preset condition, forwards the message in the first message topic to multiple second message topics, and processes the messages in the multiple second message topics in parallel. This can efficiently utilize machine configuration resources when the number of message processing nodes is fixed, thereby reducing machine cost consumption and increasing the utilization rate of machine configuration resources. Moreover, since the message processing node can process the messages in multiple second message topics in parallel, it is possible to quickly process messages and ensure the timeliness of the messages.

[0121] In order to implement the above-mentioned embodiments, the present application also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the message processing method proposed in any of the aforementioned embodiments of the present application.

[0122] Among them, the electronic device can be a personal computer (PC for short), a cloud device, a mobile device, a server, etc. The mobile device can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a car-mounted device, or any other hardware device, and this application does not impose any restrictions on this.

[0123] In order to implement the above embodiments, the present application also proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the message processing method proposed in any of the above embodiments of the present application.

[0124] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the message processing method proposed in any of the above embodiments of the present application.

[0125] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0126] like Fig.10 As shown, the electronic device 1000 may include a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0127] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as message processing methods. For example, in some embodiments, the message processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the message processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the message processing method in any other appropriate manner (e.g., by means of firmware).

[0129] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0131] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0134] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may be a cloud server, a server for a distributed system, or a server combined with a blockchain.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A message processing method, characterized in that: Applied to a message processing node, the method comprises: Obtaining processing status information of a message in a first message topic processed by the message processing node, wherein the message processing node processes the message in the first message topic in a single-topic processing mode and a multi-topic processing mode, and the single-topic processing mode is that the message processing node directly processes the message in the first message topic; In the case where it is determined that the processing status information satisfies a first preset condition, the message in the first message topic is randomly forwarded to multiple second message topics, wherein the first preset condition is a condition satisfied by the processing status information when switching from the single-topic processing mode to the multi-topic processing mode, and the number of the second message topics is determined according to the number of the message processing nodes and / or the speed at which the messages in the first message topic processed by the message processing nodes increase; The messages in a plurality of the second message topics are processed in parallel.

2. The method according to claim 1, characterized in that The processing status information includes message backlog; The step of forwarding the message in the first message topic to multiple second message topics when it is determined that the processing status information satisfies a first preset condition includes: When it is determined that the message backlog is greater than a first preset threshold, the messages in the first message topic are randomly forwarded to multiple second message topics.

3. The method according to claim 2, characterized in that Randomly forwarding the message in the first message topic to multiple second message topics, including: Obtaining a preset number of messages from the first message topic at a preset time interval; The preset number of messages obtained each time are randomly forwarded to any one of the plurality of second message topics.

4. The method according to claim 1, characterized in that: The messages in the plurality of second message topics are processed in parallel, including: Pulling messages from the corresponding second message topics in parallel through the message pulling threads corresponding to the second message topics; The messages pulled from each of the second message topics are processed in parallel.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When it is determined that the processing status information satisfies a second preset condition, forwarding the messages in the first message topic to the plurality of second message topics is stopped, and the messages in the first message topic are processed.

6. The method according to claim 5, characterized in that The processing status information includes the message backlog and the message forwarding volume within a preset time period; The step of stopping forwarding the message in the first message topic to the plurality of second message topics when it is determined that the processing status information satisfies a second preset condition includes: When it is determined that the message backlog is not greater than the first preset threshold and the message forwarding volume within the preset time period is less than the second preset threshold, stop forwarding the messages in the first message topic to the plurality of second message topics.

7. A message processing device, characterized in that: Applied to a message processing node, the device comprises: an acquisition module, used to acquire processing status information of a message in a first message topic processed by the message processing node, wherein the message processing node processes the message in the first message topic in a single-topic processing mode and a multi-topic processing mode, and the single-topic processing mode is that the message processing node directly processes the message in the first message topic; a forwarding module, configured to randomly forward messages in the first message topic to a plurality of second message topics when it is determined that the processing status information satisfies a first preset condition, wherein the first preset condition is a condition satisfied by the processing status information when switching from the single-topic processing mode to the multi-topic processing mode, and the number of the second message topics is determined according to the number of the message processing nodes and / or a speed at which messages in the first message topic processed by the message processing nodes increase; The first processing module is used to process the messages in multiple second message topics in parallel.

8. The device according to claim 7, characterized in that The processing status information includes message backlog; The forwarding module comprises: The forwarding unit is used to randomly forward the messages in the first message topic to multiple second message topics when it is determined that the message backlog is greater than a first preset threshold.

9. The device according to claim 8, characterized in that The forwarding unit includes: An acquisition subunit, configured to acquire a preset number of messages from the first message topic at a preset time interval; The forwarding subunit is used to randomly forward the preset number of messages obtained each time to any one of the multiple second message topics.

10. The device according to claim 7, characterized in that The first processing module comprises: A pulling unit, configured to pull messages from the corresponding second message topics in parallel through the message pulling threads corresponding to the second message topics; The first processing unit is used to process the messages pulled from each of the second message topics in parallel.

11. The device according to any one of claims 7 to 10, characterized in that: The device also includes: The second processing module is used to control the forwarding module to stop forwarding the messages in the first message topic to multiple second message topics and process the messages in the first message topic when it is determined that the processing status information meets the second preset condition.

12. The device according to claim 11, characterized in that The processing status information includes the message backlog and the message forwarding volume within a preset time period; The second processing module comprises: The second processing unit is used to control the forwarding module to stop forwarding the messages in the first message topic to multiple second message topics when it is determined that the message backlog is not greater than the first preset threshold and the message forwarding volume within the preset time period is less than the second preset threshold.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A message processing system, characterized in that: include: A message generator, a server cluster, and a message processor, wherein at least one message topic is distributed in the server cluster; wherein: The message generator is used to generate a message, the message having a message subject, and is used to publish the message to a corresponding message subject in a server cluster; The server cluster is used to receive the message published by the message generator and store the message in the corresponding message topic; The message processing party includes at least one message processing node, and the message processing node is used to execute the method described in any one of claims 1-6 to process the message in the first message topic in the at least one message topic.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

Images