Message processing method, system, device and equipment and storage medium

Abstract

The invention discloses a message processing method, system and device, equipment and a storage medium. Specifically, the method comprises the following steps: in response to a message reading request which is sent by a target subscription child node and carries a target node identifier of a target subscription node, shunting a plurality of message distribution nodes based on a plurality of message transmission units to obtain at least one to-be-distributed message transmission unit of each message distribution node; the target subscription node is any subscription sub-node under any subscription node in a plurality of subscription nodes under the target message subscriber, the plurality of subscription nodes are different in readable messages, and the plurality of subscription sub-nodes under the same subscription node are the same in readable messages; concurrently calling a plurality of message distribution nodes to extract a target message of a message transmission unit corresponding to the target node identifier; and feeding back a target distribution message integrated by the target messages extracted by the plurality of message distribution nodes to the target subscription child node. By using the scheme provided by the invention, the efficiency of message attribution calculation and message reading can be improved.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a message processing method, system, apparatus, device, and storage medium. Background Technology

[0002] In a typical message queue system, each consumer group can consume all message data, but each message can only be processed by a single subscriber node within the consumer group. Therefore, when multiple child nodes under a single subscription node want to subscribe to the same message, it is usually necessary to configure a message group for each child node. Due to message mutual exclusion between different subscription nodes, child nodes under different subscription nodes typically share a message group. However, the efficiency and accuracy of multiple message groups collaboratively calculating message attribution are low, and it cannot guarantee that child nodes under the same subscription node can subscribe to the same message. For example, in a system with two subscription nodes A and B, subscription node A has two child nodes a1 and a2, and subscription node B has two child nodes b1 and b2, then a1 and b1 need to share a consumer group 1, and a2 and b2 need to share a consumer group 2. Now, if the message publisher publishes a set of messages 123456, it is possible that consumer group 1 will assign messages 123 to child node a1 and messages 456 to child node b1, while consumer group 2 will assign messages 456 to child node a1 and messages 123 to child node b1. Furthermore, the creation process for message groups is complex and costly. Summary of the Invention

[0003] This application provides a message processing method, system, apparatus, device, and storage medium, which can improve the efficiency of message attribution calculation and message reading, improve the accuracy of message attribution calculation, ensure that multiple child nodes under the same subscription node can subscribe to the same message, and reduce system deployment costs. The technical solution of this application is as follows:

[0004] On the one hand, a message processing method is provided, the method comprising:

[0005] In response to a message read request sent by a target sub-node in the target message subscriber, multiple message distribution nodes are split into units based on multiple message delivery units to obtain at least one message delivery unit to be distributed by each message distribution node; the message read request carries the target node identifier of the target subscription node where the target sub-node is located; the target subscription node is any one of the multiple subscription nodes under the target message subscriber, the target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node, the multiple subscription nodes can read different messages, and the multiple subscription sub-nodes under the same subscription node can read the same messages;

[0006] Concurrently invoke the multiple message distribution nodes to extract the target message of the message delivery unit corresponding to the target node identifier from at least one message delivery unit to be distributed by itself;

[0007] The target messages extracted by each of the multiple message distribution nodes are integrated to obtain the target distribution message corresponding to the target node identifier;

[0008] The target distribution message is fed back to the target subscription child node.

[0009] On the other hand, a message processing system is provided, comprising: a target message subscriber, a subscription access node, and multiple message distribution nodes. The target message subscriber includes multiple subscription nodes, and each subscription node includes multiple subscription sub-nodes. The multiple subscription nodes can read different messages, while multiple subscription sub-nodes under the same subscription node can read the same messages. Specifically:

[0010] The target subscription sub-node is used to send a message read request to the subscription access node. The message read request carries the target node identifier of the target subscription node where the target subscription sub-node is located, and receives the target distribution message fed back by the subscription access node. The target subscription node is any one of the plurality of subscription nodes, and the target subscription sub-node is any one of the plurality of subscription sub-nodes under the target subscription node.

[0011] The subscription access node is used to perform unit splitting on the multiple message distribution nodes based on multiple message transmission units, to obtain at least one message transmission unit to be distributed by each message distribution node; to send message acquisition requests to the multiple message distribution nodes simultaneously; and to integrate the target messages extracted by the multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier.

[0012] Each message distribution node is configured to, in response to the message acquisition request, extract the target message of the message delivery unit corresponding to the target node identifier from at least one message delivery unit to be distributed by itself, and feed back the target message extracted by itself to the subscription access node.

[0013] On the other hand, a message processing apparatus is provided, the apparatus comprising:

[0014] The first unit splitting module is used to respond to message read requests sent by target subscription sub-nodes in the target message subscriber. Based on multiple message passing units, it splits multiple message distribution nodes into units to obtain at least one message passing unit to be distributed by each message distribution node. The message read request carries the target node identifier of the target subscription node where the target subscription sub-node is located. The target subscription node is any one of the multiple subscription nodes under the target message subscriber, and the target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node. The multiple subscription nodes can read different messages, and the multiple subscription sub-nodes under the same subscription node can read the same messages.

[0015] The first concurrent invocation module is used to concurrently invoke the multiple message distribution nodes to extract the target message of the message transmission unit corresponding to the target node identifier from at least one message transmission unit to be distributed by itself;

[0016] The message integration module is used to integrate the target messages extracted by the multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier.

[0017] The message feedback module is used to send the target distribution message back to the target subscription sub-node.

[0018] On the other hand, a message processing device is provided, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the message processing method as described above.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the message processing method as described above.

[0020] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the message processing method described above.

[0021] The message processing method, system, apparatus, device, and storage medium provided in this application have the following technical advantages:

[0022] In message processing applications, this application addresses the issue of multiple subscription nodes in a message processing system being able to read different messages. Multiple sub-subscribing nodes under the same subscription node use the target node identifier of their respective subscription node as the basis for message reading, ensuring that sub-subscribing nodes under the same subscription node can read the same message. The target sub-subscribing node sends a message reading request carrying the target node identifier of its target subscription node to the subscription access node. The subscription access node responds to the message reading request by performing unit-based message distribution on multiple message delivery units, obtaining at least one message delivery unit to be distributed by each message distribution node. Multiple message distribution nodes concurrently extract the target message that the target sub-subscribing node needs to read, and then integrate the target messages extracted by each message distribution node and feed them back to the target sub-subscribing node. This improves the efficiency of message attribution calculation and message reading, enhances the accuracy of message attribution calculation, ensures that multiple sub-nodes under the same subscription node can subscribe to the same message, and reduces system deployment costs. Attached Figure Description

[0023] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0025] Figure 2 This is a flowchart illustrating a message processing method provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of a data format for a message transmission unit provided in an embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating another message processing method provided in an embodiment of this application;

[0028] Figure 5 This is a flowchart illustrating another message processing method provided in an embodiment of this application;

[0029] Figure 6 This is a flowchart illustrating another message processing method provided in an embodiment of this application;

[0030] Figure 7 This is a flowchart illustrating a message distribution node load balancing process provided in an embodiment of this application;

[0031] Figure 8 This is a flowchart illustrating a message micro-batch writing process provided in an embodiment of this application;

[0032] Figure 9 A flowchart illustrating a message processing method with a subscription access node as the execution subject, provided as an embodiment of this application;

[0033] Figure 10 A flowchart illustrating another message processing method provided in an embodiment of this application;

[0034] Figure 11 This is a flowchart illustrating another message processing method provided in an embodiment of this application;

[0035] Figure 12 This is a schematic diagram of the structure of a message processing system provided in an embodiment of this application;

[0036] Figures 13a-13c This is a logic flowchart based on a message processing system provided in an embodiment of this application;

[0037] Figure 14 This is a block diagram of a message processing device provided in an embodiment of this application;

[0038] Figure 15 This is a schematic diagram of the structure of a message processing device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0040] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0041] It is understood that in the specific implementation of this application, data such as user information are involved. When the following embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0042] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application. The application environment may include multiple message subscribers, a subscription access node 20, and multiple message distribution nodes 30. A target message subscriber 10 (any one of the multiple message subscribers) includes multiple subscription nodes, and a target subscription node 11 (any one of the multiple subscription nodes) includes multiple subscription sub-nodes. The messages that can be read by the multiple subscription nodes are different, while the messages that can be read by the multiple subscription sub-nodes under the same subscription node are the same. The subscription access node 20 can establish communication connections with the multiple message subscribers and the multiple message distribution nodes 30 respectively. A target subscription sub-node 12 (any one of the multiple subscription sub-nodes) sends a message read request carrying the target node identifier of the target subscription node 11 to the subscription access node 20. In response to a message read request, the subscription access node 20 performs unit-based message distribution on multiple message distribution nodes 30, obtaining at least one message transmission unit to be distributed by each message distribution node 30. Then, it concurrently calls multiple message distribution nodes 30 to extract the target message corresponding to the aforementioned target node identifier from their own at least one message transmission unit to be distributed. Finally, it integrates the target messages extracted by each of the multiple message distribution nodes 30 to obtain the target distribution message corresponding to the target node identifier, and sends the target distribution message back to the target subscription sub-node 12. The target subscription sub-node 12 consumes the target distribution message. It should be noted that... Figure 1 This is just one example.

[0043] A message subscriber can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. A message subscriber may include network communication units, processors, and memory, etc.

[0044] Subscription access nodes can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Subscription access nodes may include network communication units, processors, and memory, among other things.

[0045] A message distribution node can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. A message distribution node may include network communication units, processors, and memory, among other things.

[0046] The aforementioned multiple message subscribers, subscription access nodes 20, and multiple message distribution nodes 30 can be used to build a message processing system, which can be a distributed system.

[0047] The following describes a specific embodiment of a message processing method provided in this application. Figure 2 This is a flowchart illustrating a message processing method provided in an embodiment of this application. This application provides the operational steps of the method described in the embodiment or flowchart, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiment is merely one possible execution order among many steps and does not represent the only execution order. In actual system or product execution, the method can be executed sequentially according to the embodiment or the accompanying drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0048] S201, the target subscription sub-node in the target message subscriber sends a message read request to the subscription access node. The message read request carries the target node identifier of the target subscription node where the target subscription sub-node is located. The target subscription node is any one of the multiple subscription nodes under the target message subscriber, and the target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node. The messages that can be read by the multiple subscription nodes are different, and the messages that can be read by the multiple subscription sub-nodes under the same subscription node are the same.

[0049] In the embodiments of this specification, the message processing system may include multiple message subscribers, each of which can subscribe to (read) all messages in the message processing system. The target message subscriber can be any one of the multiple message subscribers. Each message subscriber includes multiple subscription nodes. The multiple subscription nodes under the same message subscriber can read different messages, that is, each message in the full message can only be read by one subscription node under the same message subscriber. The target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node. Each subscription node includes multiple subscription sub-nodes. The multiple subscription sub-nodes under the same subscription node can read the same messages, that is, any subscription sub-node under the same subscription node can read the messages belonging to its own subscription node in the full message. The target subscription node is any one of the multiple subscription nodes under the target message subscriber.

[0050] In a specific embodiment, the target node identifier is used to identify the target subscription node where the target subscription child node is located. Specifically, the type of the target node identifier may include, but is not limited to, node number, node name, etc.

[0051] In one specific embodiment, a message read request can be used to request messages that can be read by the target subscription node where the target subscription child node is located.

[0052] S202, the subscription access node responds to the message reading request by performing unit splitting on multiple message distribution nodes based on multiple message transmission units, thereby obtaining at least one message transmission unit to be distributed by each message distribution node.

[0053] In the embodiments of this specification, the message passing unit can be the basic unit for processing data in the message processing system. Each message passing unit can contain multiple messages. Specifically, the data length of the message passing unit is determined by the preset maximum message storage time and the preset maximum number of messages.

[0054] In a specific embodiment, the data format of a message passing unit may include: the latest progress data of the current message passing unit and the message sequence in the current message passing unit. Specifically, the latest progress data may represent the position of the latest message written to the current message passing unit within the current message passing unit, and the latest progress data may be the maximum offset of the current message passing unit; the message sequence may be obtained by arranging the messages written to the current message passing unit in an ordered manner according to the message offsets, and the message offsets can be used to locate the position of the message within its respective message passing unit. (Illustrative example, such as...) Figure 3As shown, the data format of a message passing unit can include P:unit and D:unit, where unit represents the message passing unit, P:unit represents the maximum offset (max_offset) of the current message passing unit, the maximum offset is a key-value pair structure, and D:unit represents the actual message sequence in the current message passing unit. The message sequence can be a sorted set structure, where each message in the message sequence is associated with an offset, and each message is sorted by the offset.

[0055] In one specific embodiment, the subscription access node can be used to distribute multiple message delivery units to different message distribution nodes. Specifically, the subscription access node performs hash processing based on the unit ID of each message delivery unit and the number of nodes of the multiple message distribution nodes, with each message distribution node having at least one message delivery unit to be distributed.

[0056] In one specific embodiment, the subscription access node can obtain the number of multiple message distribution nodes from the metadata management node. Specifically, the metadata management node is used to store various metadata in the message processing system. Illustratively, the metadata management node can store the number of multiple message distribution nodes, the number of subscription nodes under each message subscriber, etc. This application does not specifically limit the data storage type of the metadata management node. Illustratively, the data storage type of the metadata management node can include, but is not limited to, Zookeeper, Etcd, Consul, etc. Specifically, the number of message distribution nodes and the number of subscription nodes under the target message subscriber are not related; the number of message distribution nodes and the number of subscription nodes can be set separately according to the message processing needs in the actual application.

[0057] It is understood that in a message processing system, when the number of message distribution nodes is updated (e.g., a new message distribution node is added or an existing message distribution node is deleted), the number of at least one message delivery unit to be distributed by the existing message distribution nodes will also be updated accordingly. Therefore, whenever a subscription access node receives a message read request from any subscription child node, it can obtain the current number of message distribution nodes from the metadata management node, and then reallocate the number of at least one message delivery unit to be distributed by each message distribution node according to the current number.

[0058] In one specific embodiment, hashing is performed based on the unit identifier (unitID) of each message delivery unit and the number of nodes of multiple message distribution nodes. The at least one message delivery unit to be distributed by each message distribution node may include: obtaining the first node mapping parameter corresponding to the unit identifier (unitID) of each message delivery unit through a hash algorithm, determining the message distribution node corresponding to the first node mapping parameter based on the pre-established first node mapping relationship, thereby determining the message distribution node corresponding to each message delivery unit. The first node mapping relationship is used to indicate the correspondence between each message distribution node and each node mapping parameter in the multiple message distribution nodes, and based on the message distribution node corresponding to each message delivery unit, determining at least one unit identifier to be distributed (i.e., the unit identifier of each of the at least one message delivery unit to be distributed) corresponding to each message distribution node.

[0059] In one specific embodiment, obtaining the first node mapping parameter corresponding to the unit identifier of each message delivery unit through a hash algorithm may include: performing a modulo operation on the unit identifier of each message delivery unit based on a preset value, and using the result of the modulo operation as the first node mapping parameter. Optionally, the preset value may be the number of nodes in multiple message distribution nodes, or it may be the modulo value in the consistent hashing algorithm, where the modulo value in the consistent hashing algorithm may be 2. 32 .

[0060] In a specific embodiment, when the first node mapping parameter is obtained by performing a modulo operation on the modulo value in the consistent hash algorithm, the subscription access node can mark each message distribution node on the hash ring based on the node mapping parameter in the first node mapping relationship. At the same time, the first node mapping parameter corresponding to the unit identifier of each message transmission unit is also marked on the hash ring. Starting from the marked position of the first node mapping parameter, the search is performed on the hash ring in a clockwise direction. The message distribution node corresponding to the first node mapping parameter found is taken as the message distribution node corresponding to each message transmission unit (i.e., the message distribution node that distributes the corresponding message transmission unit).

[0061] In a specific embodiment, the subscription access node can also perform authentication processing on the message reading request to obtain the authentication result. If the authentication result indicates that the authentication is successful, the node performs unit splitting on multiple message distribution nodes based on multiple message transmission units to obtain at least one message transmission unit to be distributed by each message distribution node. This application does not specifically limit the authentication method here.

[0062] In one specific embodiment, both the subscription access node and the message distribution node adopt stateless services. A stateless service is a service that does not need to save any data state or maintain any session information. Since stateless services can be deployed or deleted at any time without affecting the availability of the system, the subscription access node and the message distribution node are scalable.

[0063] S203, the subscription access node concurrently calls multiple message distribution nodes to extract the target message of the message distribution unit corresponding to the target node identifier from at least one message distribution unit to be distributed by itself.

[0064] In one specific embodiment, the above-mentioned subscription access node concurrently calling multiple message distribution nodes to extract the target message of the message delivery unit corresponding to the target node identifier from at least one message delivery unit to be distributed by itself may include:

[0065] S2031, the subscription access node sends message retrieval requests to multiple message distribution nodes at the same time. The message retrieval request corresponding to each message distribution node carries the target node identifier and at least one unit identifier to be distributed corresponding to each message distribution node.

[0066] S2032, each message distribution node responds to the message retrieval request by reading the message of the message delivery unit corresponding to at least one unit identifier to be distributed from the local cache area.

[0067] Specifically, each message distribution node can store messages in its local cache area, which are periodically retrieved from the message storage node and are to be distributed by at least one message delivery unit.

[0068] S2033, each message distribution node performs hash processing based on at least one unit identifier to be distributed and the number of nodes subscribed in the target message subscriber to determine the target unit identifier corresponding to the target node identifier among at least one unit identifier to be distributed.

[0069] Specifically, each message distribution node can obtain the number of subscribed nodes in the target message subscriber from the metadata management node.

[0070] Specifically, each message distribution node can obtain at least one second node mapping parameter corresponding to each of the to-be-distributed unit identifiers through a hash algorithm. Based on the pre-established second node mapping relationship, the node identifier corresponding to the second node mapping parameter is determined, thereby determining the node identifier corresponding to each to-be-distributed unit identifier. The second node mapping relationship is used to indicate the correspondence between the node identifiers of each subscription node in the target message subscriber and each node mapping parameter. Based on the node identifier corresponding to each to-be-distributed unit identifier, the target unit identifier corresponding to the target node identifier is determined.

[0071] In one specific embodiment, obtaining the second node mapping parameter corresponding to each of the at least one to-be-distributed unit identifiers using a hash algorithm may include: performing a modulo operation on each to-be-distributed unit identifier based on a preset value, and using the modulo operation result as the second node mapping parameter. Optionally, the preset value may be the number of subscribed nodes in the target message subscriber, or it may be a modulo value in the consistent hashing algorithm, where the modulo value in the consistent hashing algorithm may be 2. 32 .

[0072] In a specific embodiment, when the second node mapping parameter is obtained by performing a modulo operation on the modulo value in the consistent hash algorithm, each message distribution node can mark the node identifier of each subscription node on the hash ring based on the node mapping parameter in the second node mapping relationship. At the same time, the second node mapping parameter corresponding to each unit identifier to be distributed is also marked on the hash ring. Starting from the marked position of the second node mapping parameter, the node is searched in a clockwise direction on the hash ring. The node identifier corresponding to the first node mapping parameter found is used as the node identifier corresponding to each unit identifier to be distributed (that is, the subscription node that can read the message transmission unit corresponding to each unit identifier to be distributed is determined).

[0073] S2034, each message distribution node feeds back the target message of the message transmission unit corresponding to the target unit identifier to the subscription access node.

[0074] S204, the subscription access node integrates the target messages extracted by multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier.

[0075] Specifically, the subscription access node blocks and waits for the target messages from multiple message distribution nodes, and integrates the target messages from multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier.

[0076] In one specific embodiment, the target distribution message corresponding to the target node identifier can be a message that the target subscription node corresponding to the target node identifier can read from the full messages of the message processing system.

[0077] S205, the subscription access node sends the target distribution message back to the target subscription child node.

[0078] In an optional embodiment, the message read request sent by the target subscription sub-node may also carry multiple readable unit identifiers corresponding to the target subscription node. These multiple readable unit identifiers may refer to the unit identifiers of each of the multiple message transmission units that the target subscription node can read among all message transmission units. Correspondingly, the subscription access node can feed back a target distribution message corresponding to these multiple readable unit identifiers to the target subscription sub-node. It is understood that in the message processing system, when the number of subscription nodes in the message subscriber's party is updated (e.g., a new subscription node is added or an existing subscription node is deleted), at least one message transmission unit that the target subscription node can read will also be updated accordingly. That is, the message distribution node will reallocate at least one message transmission unit that the target subscription node can read based on a hash algorithm and the current number of subscription nodes in the target message subscriber's party. Therefore, when the number of subscription nodes is updated, the target unit identifier fed back by the message distribution node may be inconsistent with the multiple readable unit identifiers carried in the progress read request sent by the target subscription sub-node. The target subscription sub-node can update its own stored multiple readable unit identifiers based on the target unit identifier fed back by the message distribution node.

[0079] As can be seen from the above embodiments, multiple subscription nodes in the message processing system can read different messages. Multiple sub-subscribing nodes under the same subscription node use the target node identifier of their respective subscription node as the basis for message reading, enabling sub-subscribing nodes under the same subscription node to read the same message. The target sub-subscribing node sends a message reading request carrying the target node identifier of the target subscription node to the subscription access node. In response to the message reading request, the subscription access node performs unit distribution on multiple message delivery units, obtaining at least one message delivery unit to be distributed by each message distribution node. Multiple message distribution nodes concurrently extract the target message that the target sub-subscribing node needs to read, and integrate the target messages extracted by each of the multiple message distribution nodes and feed them back to the target sub-subscribing node. This can improve the efficiency of message attribution calculation and message reading, improve the accuracy of message attribution calculation, ensure that multiple sub-nodes under the same subscription node can subscribe to the same message, and reduce system deployment costs.

[0080] In a specific embodiment, before the target subscription sub-node in the aforementioned target message subscriber sends a message read request to the subscription access node, the target subscription sub-node may also send a progress read request to the subscription access node to obtain the latest message progress data of its readable message delivery unit. Based on the comparison between the latest message progress data and its current read progress data, it determines whether a new message has been generated in its readable message delivery unit, and then sends a message read request for the new message to the subscription access node. Specifically, as follows... Figure 4 As shown, the above method may further include:

[0081] S206, the target subscription child node sends a progress read request to the subscription access node.

[0082] Specifically, a progress read request can carry the target node identifier of the target subscription node and multiple readable unit identifiers corresponding to the target subscription node. Accordingly, the progress read request can be used to request the latest message progress data corresponding to each of the multiple readable unit identifiers.

[0083] S207, the subscription access node responds to the progress reading request and performs unit splitting on multiple message distribution nodes based on multiple message transmission units, so as to obtain at least one message transmission unit to be distributed by each message distribution node.

[0084] Specifically, the detailed content of "the subscription access node responds to the progress read request, performs unit splitting on multiple message distribution nodes based on multiple message transmission units, and obtains at least one message transmission unit to be distributed by each message distribution node" is similar to the detailed content of "the subscription access node responds to the message read request, performs unit splitting on multiple message distribution nodes based on multiple message transmission units, and obtains at least one message transmission unit to be distributed by each message distribution node" in step S202. For details, please refer to the detailed content of step S202, which will not be repeated here.

[0085] S208, the subscription access node concurrently calls multiple message distribution nodes to extract the latest progress data of the message transmission unit corresponding to the target node identifier from the latest progress data of at least one message transmission unit to be distributed by itself.

[0086] In one specific embodiment, the above-mentioned subscription access node concurrently calling multiple message distribution nodes to extract the target latest progress data of the message delivery unit corresponding to the target node identifier from the latest progress data of at least one message delivery unit to be distributed by itself may include:

[0087] S2081, the subscription access node simultaneously sends progress retrieval requests to multiple message distribution nodes. The message retrieval request corresponding to each message distribution node carries the target node identifier and at least one unit identifier to be distributed corresponding to each message distribution node.

[0088] S2082, each message distribution node responds to the progress acquisition request by reading the latest progress data of the message delivery unit corresponding to at least one unit identifier to be distributed from the local cache area.

[0089] Specifically, each message distribution node can store the latest progress data of at least one message delivery unit to be distributed, which is periodically obtained from the message storage node, in its local cache area.

[0090] S2083, each message distribution node performs hash processing based on at least one unit identifier to be distributed and the number of nodes subscribed in the target message subscriber to determine the target unit identifier corresponding to the target node identifier among at least one unit identifier to be distributed.

[0091] S2084, each message distribution node feeds back the latest progress data of the target unit corresponding to the target unit identifier to the subscription access node.

[0092] Specifically, the detailed content of steps S2081 to S2084 is similar to that of steps S2031 to S2034, and will not be repeated here.

[0093] S209, the subscription access node integrates the latest progress data of the target extracted by multiple message distribution nodes to obtain the latest message progress data corresponding to the target node identifier.

[0094] Specifically, the subscription access node blocks and waits for the latest progress data of the target from multiple message distribution nodes, and integrates the latest progress data of the target from multiple message distribution nodes to obtain the latest message progress data corresponding to the target node identifier.

[0095] In a specific embodiment, the latest message progress data corresponding to the target node identifier may include: the latest message progress of each of the multiple target unit identifiers corresponding to the target node identifier. It can be understood that the multiple target unit identifiers corresponding to the target node identifier may refer to the unit identifier of the message delivery unit currently allocated to the target subscription node. Specifically, the multiple target unit identifiers may include: the target unit identifiers corresponding to the target node identifiers determined by the multiple message distribution nodes mentioned above.

[0096] S210, the subscription access node sends the latest message progress data back to the target subscription child node.

[0097] S211, the target sub-node sends a message read request to the subscribed access node based on the comparison between the latest message progress data and its own current read progress data.

[0098] In one specific embodiment, the current read progress data of the target subscription sub-node itself may include: multiple readable unit identifiers, each corresponding to its current read progress. The current read progress corresponding to each readable unit identifier can be used to represent the message read progress of the target subscription sub-node for each readable unit identifier corresponding to the message delivery unit.

[0099] In an optional embodiment, the target subscription sub-node can compare the latest message progress data with its own current reading progress data. Optionally, the target subscription sub-node can compare the current reading progress of a certain readable unit identifier with the latest message progress of that readable unit identifier. When the latest message progress is greater than the current reading progress, it sends a message reading request for that readable unit identifier to the subscription access node. The target subscription sub-node can also compare multiple target unit identifiers and multiple readable unit identifiers. When there is a target unit identifier among the multiple target unit identifiers that is inconsistent with any readable unit identifier, the inconsistent target unit identifier can be used as a new readable unit identifier, and a message reading request for the new readable unit identifier can be sent to the subscription access node. Since the target subscription sub-node does not have the current reading progress corresponding to the new readable unit identifier locally, it can choose to start reading messages from the latest message progress or start reading all known messages from progress 0. This application does not impose any special limitations on this.

[0100] As can be seen from the above embodiments, the target subscription sub-node sends a progress read request to the subscription access node. In response to the progress read request, the subscription access node concurrently calls multiple message distribution nodes to extract the latest progress data of the message transmission unit corresponding to the target node identifier from the latest progress data of at least one message transmission unit to be distributed. This can reduce the computational load of a single message distribution node, thereby improving the efficiency of extracting the latest progress data. Based on the comparison result between the latest message progress data fed back by the subscription access node and its own current read progress data, the target subscription sub-node sends a message read request to the subscription access node. This can improve the targeting of the message read request and thus improve the message processing efficiency of the message processing system.

[0101] In one specific embodiment, although multiple sub-subscribing nodes under the same subscribing node can read the same messages, the message processing progress of different sub-subscribing nodes under the same subscribing node may be different. Therefore, the message reading request sent by the target sub-subscribing node also carries the current reading progress data of the target sub-subscribing node. Specifically, for example... Figure 5As shown, after the subscription access node responds to the message read request and performs unit-based traffic distribution to multiple message distribution nodes based on multiple message passing units, the above method may further include:

[0102] S501, the subscription access node concurrently calls multiple message distribution nodes to extract the new unread messages of the target subscription child node from the message distribution unit corresponding to the target node identifier in its own cache, based on the current reading progress data and the latest progress data of at least one message delivery unit cached by itself.

[0103] Specifically, as detailed in step S211 above, the target subscription sub-node can compare the current reading progress of a certain readable unit identifier with the latest message progress of that readable unit identifier. When the latest message progress is greater than the current reading progress, it sends a message reading request carrying the current reading progress of the readable unit identifier to the subscription access node. Accordingly, the newly added unread messages can include: newly added messages in the message delivery unit corresponding to the readable unit identifier (i.e., messages between the current reading progress and the latest message progress). The target subscription sub-node can also compare multiple target unit identifiers and multiple readable unit identifiers. When there is a target unit identifier among the multiple target unit identifiers that is inconsistent with any readable unit identifier, it can use the inconsistent target unit identifier as a newly added readable unit identifier and send a message reading request carrying the newly added readable unit identifier to the subscription access node. Accordingly, the newly added unread messages can include: all messages in the message delivery unit corresponding to the newly added readable unit identifier.

[0104] Specifically, the concurrent call process of the subscription access node to multiple message distribution nodes is similar to the concurrent call process of the subscription access node to multiple message distribution nodes in step S203. For details, please refer to the detailed content of step S203, which will not be repeated here.

[0105] S502, the subscription access node integrates the newly added unread messages extracted by multiple message distribution nodes to obtain the target distribution message.

[0106] As can be seen from the above embodiments, the subscription access node concurrently calls multiple message distribution nodes to extract new unread messages of the target subscription sub-node from the message distribution unit corresponding to the target node identifier based on the current reading progress data and the latest progress data of at least one message delivery unit cached by itself. The new unread messages extracted by multiple message distribution nodes are then integrated to obtain the target distribution message. This can effectively improve the targeting of message distribution nodes in extracting new unread messages, thereby improving message processing efficiency and performance and reducing the waste of system resources.

[0107] In a specific embodiment, such as Figure 6As shown, before the subscription access node integrates the target messages extracted by multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier, the above method may further include:

[0108] S601: Subscribing access nodes concurrently call multiple message distribution nodes to retrieve hot messages from message storage nodes. Hot messages are messages that can be read by multiple subscribing nodes.

[0109] Specifically, because trending messages are time-sensitive, each message distribution node needs to obtain the current trending messages from the message storage node in real time after the target subscription child node sends a message read request.

[0110] Specifically, the concurrent call process of the subscription access node to multiple message distribution nodes is similar to the concurrent call process of the subscription access node to multiple message distribution nodes in step S203. For details, please refer to the detailed content of step S203, which will not be repeated here.

[0111] Accordingly, the aforementioned subscription access node integrates the target messages extracted by multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier, which may include:

[0112] S602, the subscription access node integrates the target messages and hot messages extracted by multiple message distribution nodes to obtain the target distribution message.

[0113] As can be seen from the above embodiments, while the message distribution node extracts the target message corresponding to the target node identifier from at least one message transmission unit to be distributed, it can retain hot messages that can be read by multiple subscription nodes. The subscription access node integrates the target message and hot messages and feeds them back to the target subscription child node, so that the target subscription child node can consume both the exclusive messages of its target subscription node and the hot messages in the system, thereby improving the diversity of message consumption methods and meeting different message consumption needs.

[0114] In a specific embodiment, the at least one message delivery unit to be distributed by each message distribution node may include: multiple units to be distributed (i.e., multiple message delivery units to be distributed), and each message distribution node includes multiple distribution sub-nodes. Each distribution sub-node can be responsible for distributing multiple units to be distributed by its respective message distribution node. Specifically, each message distribution node can perform load balancing on its multiple units to be distributed based on multiple distribution sub-nodes, so that each distribution sub-node actually distributes different units to be distributed. Accordingly, such as Figure 7 As shown, the above-mentioned subscription access node concurrently calling multiple message distribution nodes to extract the target message of the message delivery unit corresponding to the target node identifier from at least one message delivery unit to be distributed by itself may include:

[0115] S701 allows subscription access nodes to concurrently call multiple message distribution nodes.

[0116] Specifically, the subscription access node can send the first message retrieval request to multiple message distribution nodes at the same time. The first message retrieval request corresponding to each message distribution node carries the target node identifier and multiple unit identifiers to be distributed corresponding to each message distribution node (i.e., the unit identifiers of the multiple units to be distributed).

[0117] S702, each message distribution node controls each of its own distribution sub-nodes through load balancing, and extracts the target message of the message delivery unit corresponding to the target node identifier from the cache area of ​​each distribution sub-node. The cache area of ​​each distribution sub-node stores messages of multiple units to be distributed from the message distribution node where each distribution sub-node is located, which are periodically obtained from the message storage node.

[0118] In one specific embodiment, each message distribution node controls its own distribution sub-nodes through load balancing. Extracting the target message of the message delivery unit corresponding to the target node identifier from the cache area of ​​each distribution sub-node may include:

[0119] S7021, in response to the first message acquisition request, the target distribution node allocates units to multiple target distribution sub-nodes based on its own multiple undistributed unit identifiers, and obtains at least one allocation unit identifier corresponding to each target distribution sub-node. The target distribution node is any message distribution node among multiple message distribution nodes, and the multiple target distribution sub-nodes are multiple distribution sub-nodes included under the target distribution node.

[0120] Specifically, the detailed content of "the target distribution node allocates units to multiple target distribution sub-nodes based on its own multiple to-be-distributed unit identifiers, and obtains at least one allocation unit identifier corresponding to each target distribution sub-node" is similar to the detailed content of "the subscription access node performs unit distribution on multiple message distribution nodes based on multiple message transmission units, and obtains at least one message transmission unit to be distributed by each message distribution node" in step S202. For details, please refer to the detailed content of step S202, which will not be repeated here.

[0121] S7022, the target distribution node simultaneously sends a second message retrieval request to multiple target distribution sub-nodes. The second message retrieval request corresponding to each target distribution sub-node carries the target node identifier and at least one allocation unit identifier corresponding to each target distribution sub-node.

[0122] S7023, each target distribution sub-node responds to the second message retrieval request by reading a message from the local cache area corresponding to the message delivery unit with at least one distribution unit identifier.

[0123] S7024, each target distribution sub-node performs hash processing based on at least one distribution unit identifier and the number of subscribed nodes in the target message subscriber to determine the target distribution unit identifier corresponding to the target node identifier among at least one distribution unit identifier.

[0124] Specifically, the detailed content of "each target distribution sub-node performs hash processing based on at least one distribution unit identifier and the number of subscribed nodes in the target message subscriber to determine the target distribution unit identifier corresponding to the target node identifier among at least one distribution unit identifier" is similar to the detailed content of step S2033. For details, please refer to the detailed content of step S2033, which will not be repeated here.

[0125] S7025, each target distribution sub-node feeds back the message of the message transmission unit corresponding to the target allocation unit identifier to the target distribution node.

[0126] S7026, the target distribution node integrates the messages extracted by multiple target distribution sub-nodes to obtain the target message.

[0127] Specifically, the target distribution node blocks and waits for messages from multiple target distribution sub-nodes, and integrates the messages from these sub-nodes to obtain the target message extracted by the target distribution node.

[0128] As can be seen from the above embodiments, each message distribution node includes multiple distribution sub-nodes. Each distribution sub-node can be responsible for distributing multiple units to be distributed by the message distribution node. After receiving concurrent call requests from the subscription access node, each message distribution node can control its own distribution sub-nodes through load balancing. The target message of the message delivery unit corresponding to the target node identifier is extracted from the cache area of ​​each distribution sub-node. The cache area of ​​each distribution sub-node stores messages of multiple units to be distributed by the message distribution node where each distribution sub-node is located, which are periodically obtained from the message storage node. This can further improve the efficiency of message attribution calculation and message reading. At the same time, message data is cached in the cache area of ​​each distribution sub-node, reducing the number of requests to the message storage node and improving the performance of the message processing system.

[0129] In one specific embodiment, the above method may further include:

[0130] 1) Each message distribution node periodically retrieves new messages from at least one message delivery unit to be distributed from the message storage node.

[0131] In one specific embodiment, each message distribution node can periodically obtain the latest progress data of at least one message delivery unit to be distributed from the message storage node, compare the latest progress data with the current progress data in the cache area, obtain the comparison result, and based on the comparison result, obtain the new message of the at least one message delivery unit from the message storage node.

[0132] 2) Each message distribution node stores newly added messages in its own cache area.

[0133] 3) Each message distribution node periodically deletes invalid messages from the cache area.

[0134] In a specific embodiment, the failed message can be a message in the cache area that meets the preset failure conditions. Specifically, the preset failure conditions can be preset in combination with the message timeliness requirements in actual applications. In an optional embodiment, the preset failure condition can be set as follows: the writing time of the message in the cache area is greater than the preset time. The preset failure condition can also be set as follows: the progress difference between the progress data of the message in the message transmission unit and the latest progress data of the message transmission unit is greater than the preset threshold.

[0135] As can be seen from the above embodiments, each message distribution node can periodically obtain new messages of at least one message delivery unit to be distributed from the message storage node, update the new messages to its own cache area, and periodically delete invalid messages in the cache area, which can improve the accuracy and effectiveness of messages stored in the cache area, thereby improving the accuracy and effectiveness of message distribution.

[0136] In a specific embodiment, such as Figure 8 As shown, the above method may further include:

[0137] S801, the message publisher sends the message to the message writer node.

[0138] Specifically, the message publisher can send the message to the publishing access node, which will then send the message to the message writing node.

[0139] In one specific embodiment, after receiving a publish message from a message publisher, the publishing access node records the corresponding WAL (Write Ahead Log) for the published message and writes the published message to its local cache area based on the WAL log. Optionally, the publishing access node can also perform authentication processing on the published message; only after the published message has been successfully authenticated will the step of recording the corresponding WAL log be executed.

[0140] S802, the message writing node writes the published message to the message storage node through micro-batch processing.

[0141] In one specific embodiment, a message writing node may include multiple write service nodes. The message writing node may write the published message to the message storage node through micro-batch processing, which may include:

[0142] S8021, the publishing access node randomly assigns a unit identifier from multiple unit identifiers for each published message.

[0143] Specifically, multiple unit identifiers can be the unit identifier corresponding to any message transmission unit among multiple message transmission units in the message processing system.

[0144] S8022, the publishing access node performs hash processing based on each unit identifier and the number of write service nodes in the message processing system to determine the write service node corresponding to each unit identifier.

[0145] Specifically, the detailed content of "the publishing access node performs hash processing based on the number of nodes writing service nodes under each unit identifier and the message writing node to determine the writing service node corresponding to each unit identifier" is similar to the detailed content of step S2033. For details, please refer to the detailed content of step S2033, which will not be repeated here.

[0146] S8022, when multiple messages to be written corresponding to the unit identifier to be written in the local cache area meet the preset batch writing conditions, the publishing access node sends a message writing request to the target writing service node corresponding to the unit identifier to be written. The message writing request carries the unit identifier to be written and multiple messages to be written. The unit identifier to be written is any one of the multiple unit identifiers.

[0147] Specifically, the preset batch write conditions can be pre-set based on the message batch processing requirements in actual applications. In an optional embodiment, the preset batch write conditions can be set as follows: the number of multiple messages to be written reaches a preset threshold, for example, the preset threshold is 100; the preset batch write conditions can also be set as follows: the time since the last write to the message transmission unit corresponding to the unit to be written reaches a preset waiting time, for example, the preset waiting time is 50ms.

[0148] S8023, the target write service node responds to the message write request by locking the message transmission unit corresponding to the identifier of the unit to be written through the distributed lock node.

[0149] Specifically, the distributed lock node is responsible for maintaining the distributed lock in the message processing system, and can execute logic such as locking, unlocking, and timeout release.

[0150] S8024, the target write service node generates progress data for each message to be written in the message delivery unit corresponding to the message unit identifier in the multiple messages to be written through the progress generation node, and updates the latest progress data corresponding to the message storage node to the message unit identifier.

[0151] Specifically, the progress generation node is responsible for managing the globally incrementing progress data for each message passing unit.

[0152] S8024, the target write service node sends multiple messages to be written corresponding to the identifier of the unit to be written to the message storage node.

[0153] S8025, the message storage node sorts multiple messages to be written in ascending order of their progress data and obtains a message sequence, and stores the message sequence in the message transmission unit corresponding to the message unit identifier to be written.

[0154] S8026, the target write service node unlocks the message passing unit corresponding to the identifier of the unit to be written through the distributed lock node, and sends a write success result back to the publishing access node.

[0155] In an optional embodiment, both the publishing access node and the message writing node adopt stateless services and can be horizontally scaled arbitrarily. The message storage node is responsible for actually storing the messages. This application does not impose specific restrictions on the storage type of the message storage node. For illustrative purposes, the storage type of the message storage node may include, but is not limited to, Redis, Etcd, MySQL, etc.

[0156] As can be seen from the above embodiments, message production based on micro-batch processing, and the automatic triggering of subsequent message writing to the message delivery unit after the access node publishes messages based on a preset writing period or accumulates a preset number of messages, can effectively improve the production and consumption speed of the message processing system.

[0157] In one specific embodiment, the write service node can also periodically delete failure information of message delivery units in the message storage node. Specifically, the write service node can obtain the current data length and earliest data time of the unit to be processed from the message storage node, determine the failure message of the unit to be processed based on the current data length and earliest data time, and request the message storage node to delete the failure information of the message delivery unit. The unit to be processed is any message delivery unit among at least one message delivery unit that the write service node is responsible for.

[0158] In an optional embodiment, determining the failure message of the unit to be processed based on the current data length and the earliest data time may include: if the current data length is greater than the preset data length, taking the message in the unit to be processed that is before the preset data length as the failure message; or, if the time difference between the current time and the earliest data time is greater than the preset storage time, taking the message in the unit to be processed that has a duration greater than the preset storage time as the failure message.

[0159] As can be seen from the technical solutions provided in the above embodiments of this application, in message processing application scenarios, message writing is performed based on micro-batch processing. After the publishing access node writes messages based on a preset writing period or accumulates a preset number of messages, the subsequent message writing nodes are automatically triggered to write the messages to the message delivery unit. This can effectively improve the production and consumption speed of the message processing system. In the message processing system, each message subscriber can read all messages. Each message subscriber includes multiple subscription nodes. The messages that multiple subscription nodes under the same message subscriber can read are different. Multiple subscription child nodes under the same subscription node use the target node identifier of the subscription node as the basis for message reading, so that the subscription child nodes under the same subscription node can read the same messages. The target subscription child node sends a message read message carrying the target node identifier of the target subscription node to the subscription access node. The system retrieves requests, and the subscription access node responds to message read requests. Based on multiple message delivery units, multiple message distribution nodes are divided into units, resulting in at least one message delivery unit to be distributed by each message distribution node. Multiple message distribution nodes concurrently extract the target message that the target sub-subscribing node needs to read, and the target messages extracted by each of the multiple message distribution nodes are integrated and fed back to the target sub-subscribing node. This can improve the efficiency of message attribution calculation and message reading, as well as the accuracy of message attribution calculation, ensuring that multiple sub-nodes under the same subscription node can subscribe to the same message. In addition, by separating storage nodes from logical computing nodes (message writing nodes and message distribution nodes), and using stateless services for logical computing nodes, the system supports cloud-native deployment and can be horizontally expanded indefinitely, improving the scalability of the message processing system.

[0160] The following describes another message processing method provided in the embodiments of this application, taking the subscription access node as the execution subject. Figure 9This is a flowchart illustrating another message processing method provided in an embodiment of this application. It should be noted that this specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as... Figure 9 As shown, the method may include:

[0161] S901, in response to a message read request sent by a target sub-node in the target message subscriber, the message distribution nodes are split into units based on multiple message delivery units to obtain at least one message delivery unit to be distributed by each message distribution node; the message read request carries the target node identifier of the target subscription node where the target sub-node is located; the target subscription node is any one of the multiple subscription nodes under the target message subscriber, the target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node, the messages that can be read by the multiple subscription nodes are different, and the messages that can be read by the multiple subscription sub-nodes under the same subscription node are the same;

[0162] S902, concurrently call multiple message distribution nodes to extract the target message of the message transmission unit corresponding to the target node identifier from at least one message transmission unit to be distributed by itself;

[0163] S903, integrate the target messages extracted by multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier;

[0164] S904 sends the target distribution message back to the target subscribed child nodes.

[0165] In one specific embodiment, before responding to the message read request sent by the target subscription child node in the target message subscriber, as follows: Figure 10 As shown, the above method may further include:

[0166] S905, in response to the progress read request sent by the target subscription sub-node, the multiple message distribution nodes are split into units based on multiple message transmission units, so as to obtain at least one message transmission unit to be distributed by each message distribution node.

[0167] S906, concurrently call multiple message distribution nodes to extract the latest progress data of the message transmission unit corresponding to the target node identifier from the latest progress data of at least one message transmission unit to be distributed by itself;

[0168] S907, integrate the latest progress data of the target extracted by multiple message distribution nodes to obtain the latest message progress data corresponding to the target node identifier;

[0169] S908, the latest message progress data is fed back to the target sub-node, so that the target sub-node can send a message reading request based on the comparison between the latest message progress data and its own current reading progress data.

[0170] In one specific embodiment, the message read request also carries the current read progress data of the target sub-node. After performing unit-based diversion on multiple message distribution nodes based on multiple message passing units to obtain at least one message passing unit to be distributed by each message distribution node, the above method may further include:

[0171] S9021, concurrently calling multiple message distribution nodes to extract new unread messages of the target sub-node from the message transmission unit to be distributed corresponding to the target node identifier based on the current reading progress data and the latest progress data of at least one message transmission unit cached by itself;

[0172] S9031 integrates the newly added unread messages extracted by multiple message distribution nodes to obtain the target distribution message.

[0173] In a specific embodiment, such as Figure 11 As shown, before integrating the target messages extracted by multiple message distribution nodes to obtain the target distribution message, the above method may further include:

[0174] S1101, concurrently call multiple message distribution nodes to obtain hot messages from message storage nodes. Hot messages are messages that can be read by multiple subscription nodes;

[0175] Accordingly, the above-mentioned integration of the target messages extracted by multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier may include:

[0176] S1102, integrate the target messages and hot messages extracted by multiple message distribution nodes to obtain the target distribution message.

[0177] In a specific embodiment, the at least one message delivery unit to be distributed by each message distribution node may include: multiple units to be distributed, and each message distribution node includes multiple distribution sub-nodes. The above-mentioned concurrent invocation of multiple message distribution nodes to extract the target message of the message delivery unit corresponding to the target node identifier from its own at least one message delivery unit to be distributed may include:

[0178] Multiple message distribution nodes are invoked concurrently, so that each message distribution node controls each of its own distribution sub-nodes through load balancing. The target message of the message delivery unit corresponding to the target node identifier is extracted from the cache area of ​​each distribution sub-node. The cache area of ​​each distribution sub-node stores messages of multiple undistributed units of the message distribution node where each distribution sub-node is located, which are periodically obtained from the message storage node.

[0179] In one specific embodiment, the above method may further include:

[0180] 1) Control each message distribution node to periodically obtain new messages of at least one message delivery unit to be distributed from the message storage node, and store the new messages in its own cache area;

[0181] 2) Control each message distribution node to periodically delete invalid messages from the cache area.

[0182] In one specific embodiment, the messages of multiple message passing units are written in batches by the publishing access node from the message publisher through micro-batch processing.

[0183] The detailed steps of the above-described embodiment of the message processing method dictated from the subscription access node can be found in the above-described embodiment of the interactive side message processing method, and will not be repeated here.

[0184] As can be seen from the technical solutions provided in the above embodiments of this application, in message processing application scenarios, message writing is performed based on micro-batch processing. After the publishing access node writes messages based on a preset writing period or accumulates a preset number of messages, the subsequent message writing nodes are automatically triggered to write the messages to the message delivery unit. This can effectively improve the production and consumption speed of the message processing system. In the message processing system, each message subscriber can read all messages. Each message subscriber includes multiple subscription nodes. The messages that multiple subscription nodes under the same message subscriber can read are different. Multiple subscription child nodes under the same subscription node use the target node identifier of the subscription node as the basis for message reading, so that the subscription child nodes under the same subscription node can read the same messages. The target subscription child node sends a message read message carrying the target node identifier of the target subscription node to the subscription access node. The system retrieves requests, and the subscription access node responds to message read requests. Based on multiple message delivery units, multiple message distribution nodes are divided into units, resulting in at least one message delivery unit to be distributed by each message distribution node. Multiple message distribution nodes concurrently extract the target message that the target sub-subscribing node needs to read, and the target messages extracted by each of the multiple message distribution nodes are integrated and fed back to the target sub-subscribing node. This can improve the efficiency of message attribution calculation and message reading, as well as the accuracy of message attribution calculation, ensuring that multiple sub-nodes under the same subscription node can subscribe to the same message. In addition, by separating storage nodes from logical computing nodes (message writing nodes and message distribution nodes), and using stateless services for logical computing nodes, the system supports cloud-native deployment and can be horizontally expanded indefinitely, improving the scalability of the message processing system.

[0185] The following describes a message processing system provided by an embodiment of this application. Specifically, the system includes: a target message subscriber, a subscription access node, and multiple message distribution nodes. The target message subscriber includes multiple subscription nodes, and each subscription node includes multiple subscription sub-nodes. The messages that can be read by the multiple subscription nodes are different, while the messages that can be read by the multiple subscription sub-nodes under the same subscription node are the same.

[0186] The target subscription sub-node is used to send a message read request to the subscription access node. The message read request carries the target node identifier of the target subscription node where the target subscription sub-node is located, and receives the target distribution message fed back by the subscription access node. The target subscription node can be any one of multiple subscription nodes, and the target subscription sub-node can be any one of multiple subscription sub-nodes under the target subscription node.

[0187] Subscribe to access nodes to perform unit distribution on multiple message distribution nodes based on multiple message transmission units, and obtain at least one message transmission unit to be distributed by each message distribution node; send message retrieval requests to multiple message distribution nodes simultaneously; integrate the target messages extracted by multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier;

[0188] Each message distribution node is used to respond to a message retrieval request by extracting the target message of the message delivery unit corresponding to the target node identifier from at least one message delivery unit to be distributed by itself, and feeding back the target message it extracted to the subscribing access node.

[0189] In one specific embodiment, the target subscription sub-node is also used to send a message read request to the subscription access node; receive the latest message progress data fed back by the subscription access node; and send a message read request based on the comparison result between the latest message progress data and its own current read progress data.

[0190] Subscribing to access nodes is also used to respond to progress reading requests, perform unit distribution on multiple message distribution nodes based on multiple message transmission units, and obtain at least one message transmission unit to be distributed by each message distribution node; send progress retrieval requests to multiple message distribution nodes simultaneously; and integrate the latest target progress data extracted by each of the multiple message distribution nodes to obtain the latest message progress data corresponding to the target node identifier.

[0191] Each message distribution node is also used to extract the latest progress data of the message delivery unit corresponding to the target node identifier from the latest progress data of at least one message delivery unit to be distributed by itself in response to a progress acquisition request.

[0192] In one specific embodiment, the message reading request also carries the current reading progress data of the target sub-node, and the subscription access node is also used to send message retrieval requests carrying the current reading progress data to multiple message distribution nodes simultaneously; the newly added unread messages extracted by the multiple message distribution nodes are integrated to obtain the target distribution message;

[0193] Each message distribution node is also used to extract new unread messages of the target sub-node from its own message delivery unit corresponding to the target node identifier, based on the current reading progress data and the latest progress data of at least one message delivery unit cached by itself.

[0194] In one specific embodiment, each message distribution node is also used to obtain hot messages from the message storage node, where hot messages are messages that can be read by multiple subscription nodes;

[0195] The subscription access node is also used to integrate the target messages and hot messages extracted by multiple message distribution nodes to obtain the target distribution message.

[0196] In one specific embodiment, at least one message delivery unit to be distributed by each message distribution node may include: multiple units to be distributed, multiple distribution sub-nodes under each message distribution node, and each message distribution node is also used to control each of its own distribution sub-nodes through load balancing.

[0197] Each distribution sub-node is also used to extract the target message of the message delivery unit corresponding to the target node identifier from its own cache area. The cache area of ​​each distribution sub-node stores messages of multiple units to be distributed from the message distribution node where each distribution sub-node is located, which are periodically obtained from the message storage node.

[0198] In one specific embodiment, each message distribution node is also used to periodically obtain new messages of at least one message delivery unit to be distributed from the message storage node, and store the new messages in its own cache area; and periodically delete invalid messages in the cache area.

[0199] In one specific embodiment, the system further includes: a message publisher, a message writer node, and a message storage node, wherein:

[0200] The message publisher is used to send published messages to the message writer nodes;

[0201] The message writing node is used to write published messages to the message storage node through micro-batch processing.

[0202] In one specific embodiment, the system further includes: a publishing access node, a distributed lock node, and a progress generation node; the message writing node includes multiple write service nodes, wherein:

[0203] The publishing access node is also used to randomly assign a unit identifier from multiple unit identifiers for each published message; based on each unit identifier and the number of write service nodes in the message processing system, a hash is performed to determine the write service node corresponding to each unit identifier; when multiple messages to be written corresponding to the unit identifier to be written in the local cache area meet the preset batch write conditions, a message write request is sent to the target write service node corresponding to the unit identifier to be written. The message write request carries the unit identifier to be written and multiple messages to be written. The unit identifier to be written is any one of the multiple unit identifiers.

[0204] The target write service node is also used to respond to message write requests by locking the message delivery unit corresponding to the write unit identifier through the distributed lock node; generating progress data for each of the multiple messages to be written in the message delivery unit corresponding to the write unit identifier through the progress generation node, and updating the latest progress data corresponding to the write unit identifier to the message storage node; sending the multiple messages to be written corresponding to the write unit identifier to the message storage node; and after the message storage node has finished storing the multiple messages to be written, unlocking the message delivery unit corresponding to the write unit identifier through the distributed lock node, and feeding back the write success result to the publishing access node.

[0205] The message storage node is also used to write the latest progress data corresponding to the unit identifier to be written; according to the progress data of each message to be written in ascending order, multiple messages to be written are sorted to obtain a message sequence, and the message sequence is stored in the message transmission unit corresponding to the unit identifier to be written.

[0206] See Figure 12 , Figure 12 This is a schematic diagram of the architecture of a message processing system provided in an embodiment of this application. Specifically, the message processing system may include: a message publisher, multiple message subscribers, and a message system. The message system may include: a publishing access node, a message writing node, a distributed lock node, a progress generation node, a metadata management node, a message storage node, a subscription access node, and multiple message distribution nodes, wherein:

[0207] Message publisher: Responsible for generating the message.

[0208] Publishing access node: The external interface for communication with message publishers. The publishing access node is responsible for message authentication, packaging and distribution of write requests, and generating write-ahead logs related to message processing.

[0209] Message writing node: Responsible for actually writing messages to message storage nodes. Specifically, the request progress generation node generates globally incrementing progress data, and the request distributed lock node locks the message passing unit before writing the message to the message passing unit in the message storage module.

[0210] Progress generation node: Responsible for managing the globally incrementing progress data for each message passing unit.

[0211] Distributed lock node: Responsible for maintaining the distributed lock and executing logic such as locking, unlocking, and timeout release.

[0212] Metadata management node: Stores metadata such as the number of write service nodes under the message write node, the number of subscribed nodes under each message subscriber, and the number of subscribed child nodes under each subscriber node.

[0213] Message storage node: Persistent message storage. The data format of message passing units in the message storage node can include: the maximum offset (max_offset) of the key-value pair structure and the message sequence of the sorted set structure.

[0214] Multiple message subscribers: Each message subscriber can read all messages in the message processing system. Each message subscriber includes multiple subscription nodes, and each subscription node includes multiple subscription child nodes. Multiple subscription nodes under the same message subscriber can read different messages, while multiple subscription child nodes under the same subscription node can read the same messages.

[0215] Subscription access node: The external interface for communication with message subscribers. The subscription access node is responsible for message authentication, packaging, and distribution of read requests.

[0216] Multiple message distribution nodes: Different message distribution nodes can be used to distribute data of different message delivery units. Each message distribution node can include multiple distribution sub-nodes. Multiple distribution sub-nodes under the same subscription node can handle the same message delivery units. At the same time, each distribution sub-node stores message data in a local cache area, reducing the number of times the message storage module is requested.

[0217] See Figure 13a , Figure 13a This application provides a flowchart of the writing logic based on the aforementioned message processing system. Specifically, the publishing access node writes messages from the message publisher to its local cache, randomly assigns a unitID (unit identifier of the message delivery unit) to each message, and calculates a consistent hash based on each unitID and the number of message writing nodes to determine the specific message writing node corresponding to each unitID. Based on a preset writing period or after accumulating a preset number of messages, the subsequent message writing process for each unitID's corresponding unit is automatically triggered. In the message writing process of each unitID's unit, the specific message writing node corresponding to each unitID locks the unit through a distributed lock node. Then, the progress generation node generates an increasing offset (progress value) for multiple messages to be written corresponding to each unitID, and updates the maximum progress value to the message storage node. The message storage node sorts the multiple messages to be written in ascending order of message offset and stores them in the unit. The specific message writing node corresponding to each unitID can also periodically delete invalid information in the message storage to ensure the validity of messages in the message storage.

[0218] See Figure 13b , Figure 13bThis application provides a flowchart of a progress reading logic based on the aforementioned message processing system. Specifically, the current sub-node of the message subscriber can send a progress reading request to the subscription access node. This progress reading request carries information about the current sub-node and readable unit information. In response to the progress reading request, the subscription access node calculates a consistent hash based on multiple unitIDs and the number of message distribution nodes in the system to determine the unit package corresponding to each message distribution node (i.e., determine the units to be distributed by each message distribution node). The subscription access node concurrently requests multiple message distribution nodes based on the information of the current sub-node. Each message distribution node calculates a consistent hash according to each unitID in its own unit package and the number of sub-nodes under the message subscriber, extracts the maximum progress value of the units that the current sub-node needs to process from its local cache, and returns the request result to the subscription access node. The subscription access node blocks and waits for the request results from all message distribution nodes, assembles the maximum progress value of all units processed by the current sub-node, and feeds back the maximum progress value of all units to the current sub-node. Furthermore, each message distribution node can also periodically and asynchronously update the maximum progress value of all units in its local cache.

[0219] See Figure 13c , Figure 13cThis application provides a flowchart of a message reading logic based on the aforementioned message processing system. Specifically, the current subscribed child node under the message subscriber can read the current progress value locally. If the latest progress value (i.e., the maximum progress value) of any unit reported by the subscription access node is greater than the current progress value of that unit, it indicates that there are new messages to be read for that unit. The current subscribed child node can send a message reading request to the subscription access node. This message reading request carries information about the current subscribed node, readable unit information, and current progress. In response to the message reading request, the subscription access node calculates a consistent hash based on multiple unitIDs and the number of message distribution nodes in the system to determine the unit package corresponding to each message distribution node (i.e., determine the units to be distributed by each message distribution node). The subscription access node concurrently requests multiple message distribution nodes based on the information of the current subscribed node. Each message distribution node obtains hot messages that can be read by all subscribed nodes from the message storage node, calculates a consistent hash according to each unitID in its own unit package and the number of subscribed nodes under the message subscriber, extracts the new messages of the units that the current subscribed node needs to process from the local cache, and returns the request result (hot messages + new unit messages) to the subscription access node. The subscription access node blocks and waits for the request results from all message distribution nodes, assembles the messages for all units processed by the currently subscribed node, and feeds back the messages for all units to the currently subscribed child nodes. In addition, each message distribution node can also periodically and asynchronously update the local cache with new messages for all units and delete expired messages in the local cache.

[0220] Specifically, the aforementioned message processing system is suitable for scenarios where message subscription services require multiple shards and replicas. It can efficiently handle message retrieval logic on the consumer side and reduce business complexity. Illustratively, this message processing system can be applied to a multimedia information management system. Multiple message subscribers can represent multiple multimedia social platforms within the system. Multiple subscription nodes under each subscriber can represent multiple social groups within each social platform. Multiple subscription sub-nodes under each subscription node can represent multiple users within each social group. The multimedia information management system adopts a read-write separation architecture. The message writing link is responsible for writing data into the message system, while the message distribution link updates the cached data in the message distribution nodes by periodically requesting the message system, supporting diverse consumption methods.

[0221] This application provides a message processing device, such as... Figure 14 As shown, the message processing device may include:

[0222] The first unit splitting module 1410 is used to respond to message read requests sent by target subscription sub-nodes in the target message subscriber, and to split multiple message distribution nodes based on multiple message passing units to obtain at least one message passing unit to be distributed by each message distribution node; the message read request carries the target node identifier of the target subscription node where the target subscription sub-node is located; the target subscription node is any one of the multiple subscription nodes under the target message subscriber, the target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node, the messages that can be read by multiple subscription nodes are different, and the messages that can be read by multiple subscription sub-nodes under the same subscription node are the same;

[0223] The first concurrent call module 1420 is used to concurrently call multiple message distribution nodes to extract the target message of the message transmission unit corresponding to the target node identifier from at least one message transmission unit to be distributed by itself.

[0224] The message integration module 1430 is used to integrate the target messages extracted by multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier.

[0225] The message feedback module 1440 is used to send target distribution messages back to the target subscription child nodes.

[0226] In one specific embodiment, the above method may further include:

[0227] The second unit splitting module is used to respond to the progress reading request sent by the target subscription sub-node, and to split multiple message distribution nodes based on multiple message transmission units to obtain at least one message transmission unit to be distributed by each message distribution node.

[0228] The second concurrent call module is used to concurrently call multiple message distribution nodes to extract the latest progress data of the message transmission unit corresponding to the target node identifier from the latest progress data of at least one message transmission unit to be distributed by itself.

[0229] The progress integration module is used to integrate the latest progress data of the target extracted by multiple message distribution nodes to obtain the latest message progress data corresponding to the target node identifier.

[0230] The progress feedback module is used to provide the latest message progress data to the target subscription child nodes, so that the target subscription child nodes can send message reading requests based on the comparison between the latest message progress data and their own current reading progress data.

[0231] In one specific embodiment, the message read request also carries the current read progress data of the target subscribed child node, and the above-mentioned device may further include:

[0232] The third concurrent call module is used to concurrently call multiple message distribution nodes based on the current reading progress data and the latest progress data of at least one message transmission unit cached by itself, and extract the new unread messages of the target sub-node from the message transmission unit to be distributed corresponding to the target node identifier.

[0233] A new unread message integration module has been added, which is used to integrate the newly added unread messages extracted by multiple message distribution nodes to obtain the target distribution message.

[0234] In one specific embodiment, the above-described apparatus may further include:

[0235] The fourth concurrent call module is used to concurrently call multiple message distribution nodes to retrieve hot messages from the message storage node. Hot messages are messages that can be read by multiple subscription nodes.

[0236] Accordingly, the aforementioned message integration module 1430 may include:

[0237] The hotspot message integration unit is used to integrate the target messages and hotspot messages extracted by multiple message distribution nodes to obtain the target distribution message.

[0238] In one specific embodiment, the at least one message delivery unit to be distributed by each message distribution node may include: multiple units to be distributed, and each message distribution node includes multiple distribution sub-nodes. The aforementioned first concurrent call module 1420 may include:

[0239] The cache retrieval unit is used to concurrently call multiple message distribution nodes, so that each message distribution node controls its own distribution sub-nodes through load balancing, and retrieves the target message of the message delivery unit corresponding to the target node identifier from the cache area of ​​each distribution sub-node. The cache area of ​​each distribution sub-node stores messages of multiple units to be distributed from the message distribution node where each distribution sub-node is located, which are periodically retrieved from the message storage node.

[0240] In one specific embodiment, the above-described apparatus may further include:

[0241] A new message acquisition module is added to control each message distribution node to periodically acquire new messages of at least one message delivery unit to be distributed from the message storage node and store the new messages in its own cache area.

[0242] The expired message deletion module is used to control each message distribution node to periodically delete expired messages from the cache area.

[0243] In one specific embodiment, the messages of multiple message passing units are written in batches by the message writing node from the message publisher using a micro-batch processing method.

[0244] It should be noted that the apparatus in the device embodiment and the method embodiment are based on the same inventive concept.

[0245] This application provides a message processing device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the message processing method provided in the above method embodiments.

[0246] Furthermore, Figure 15 A schematic diagram of the hardware structure of a message processing device for implementing the message processing method provided in the embodiments of this application is shown. The message processing device may participate in or include the message processing apparatus provided in the embodiments of this application. Figure 15 As shown, the message processing device 150 may include one or more processors 1502 (shown as 1502a, 1502b, ..., 1502n in the figure) (processor 1502 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1504 for storing data, and a transmission device 1506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 15 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the message processing device 150 may also include... Figure 15 The more or fewer components shown, or having the same Figure 15 The different configurations shown.

[0247] It should be noted that the aforementioned one or more processors 1502 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be wholly or partially integrated into any other element within the message processing device 150 (or mobile device). As involved in the embodiments of this application, the data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0248] The memory 1504 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the message processing method described in the embodiments of this application. The processor 1502 executes various functional applications and data processing by running the software programs and modules stored in the memory 1504, thereby implementing the aforementioned message processing method. The memory 1504 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1504 may further include memory remotely located relative to the processor 1502, and these remote memories can be connected to the message processing device 150 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0249] The transmission device 1506 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the message processing device 150. In one example, the transmission device 1506 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In one embodiment, the transmission device 1506 may be a radio frequency (RF) module for wireless communication with the Internet.

[0250] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows a user to interact with the user interface of the messaging device 150 (or mobile device).

[0251] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a message processing device to store at least one instruction or at least one program related to implementing the message processing method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the message processing method provided in the above method embodiment.

[0252] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0253] Embodiments of this application also provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the message processing method provided in the method embodiments.

[0254] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0255] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0256] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and apparatus embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0257] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0258] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A message processing method, characterized in that, The method includes: In response to a message read request sent by a target sub-node in the target message subscriber, multiple message distribution nodes are split into units based on multiple message delivery units to obtain at least one message delivery unit to be distributed by each message distribution node; the message read request carries the target node identifier of the target subscription node where the target sub-node is located; the target subscription node is any one of the multiple subscription nodes under the target message subscriber, the target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node, the multiple subscription nodes can read different messages, and the multiple subscription sub-nodes under the same subscription node can read the same messages; Concurrently invoke the multiple message distribution nodes to extract the target message of the message delivery unit corresponding to the target node identifier from at least one message delivery unit to be distributed by itself; The target messages extracted by each of the multiple message distribution nodes are integrated to obtain the target distribution message corresponding to the target node identifier; The target distribution message is fed back to the target subscription child node.

2. The method according to claim 1, characterized in that, The at least one message delivery unit to be distributed by each message distribution node includes: multiple units to be distributed, and each message distribution node includes multiple distribution sub-nodes. The concurrent invocation of the multiple message distribution nodes to extract the target message of the message delivery unit corresponding to the target node identifier from the at least one message delivery unit to be distributed by themselves includes: The multiple message distribution nodes are invoked concurrently, so that each message distribution node controls each of its own distribution sub-nodes through load balancing, and extracts the target message of the message delivery unit corresponding to the target node identifier from the cache area of ​​each distribution sub-node. The cache area of ​​each distribution sub-node stores messages of multiple units to be distributed from the message storage node, which are periodically obtained from the message storage node.

3. The method according to claim 1, characterized in that, Before integrating the target messages extracted by the multiple message distribution nodes to obtain the target distribution message, the method further includes: The multiple message distribution nodes are concurrently invoked to retrieve hot messages from the message storage node. The hot messages are messages that can be read by all the multiple subscription nodes. Accordingly, the step of integrating the target messages extracted by the multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier includes: The target message is obtained by integrating the target messages extracted by each of the multiple message distribution nodes and the hot messages.

4. The method according to claim 1, characterized in that, The message read request also carries the current read progress data of the target sub-node. After the method performs unit-based distribution on multiple message distribution nodes based on multiple message passing units to obtain at least one message passing unit to be distributed by each message distribution node, the method further includes: The concurrent call to the multiple message distribution nodes is based on the current reading progress data and the latest progress data of the at least one message delivery unit cached by themselves, and extracts the new unread messages of the target sub-node from the message delivery unit to be distributed corresponding to the target node identifier. The newly added unread messages extracted by each of the multiple message distribution nodes are integrated to obtain the target distribution message.

5. The method according to claim 1, characterized in that, Before responding to a message read request sent by a target subscription child node in the target message subscriber, the method further includes: In response to the progress read request sent by the target subscription sub-node, the multiple message distribution nodes are divided into units based on the multiple message delivery units to obtain at least one message delivery unit to be distributed by each message distribution node; Concurrently invoke the multiple message distribution nodes to extract the latest progress data of the message delivery unit corresponding to the target node identifier from the latest progress data of at least one message delivery unit to be distributed by itself; The latest progress data of the target extracted by each of the multiple message distribution nodes is integrated to obtain the latest message progress data corresponding to the target node identifier; The latest message progress data is fed back to the target subscription sub-node, so that the target subscription sub-node sends the message reading request based on the comparison result between the latest message progress data and its own current reading progress data.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Each message distribution node is controlled to periodically obtain new messages of at least one message delivery unit to be distributed from the message storage node, and store the new messages in its own cache area; Each message distribution node is controlled to periodically delete invalid messages from the cache area.

7. The method according to any one of claims 1 to 5, characterized in that, The messages of the multiple message passing units are written in batches by the message writing nodes from the message publishers through micro-batch processing.

8. A message processing system, characterized in that, The system includes: a target message subscriber, a subscription access node, and multiple message distribution nodes. The target message subscriber includes multiple subscription nodes, and each subscription node includes multiple subscription sub-nodes. The multiple subscription nodes can read different messages, while multiple subscription sub-nodes under the same subscription node can read the same messages. The target subscription sub-node is used to send a message read request to the subscription access node. The message read request carries the target node identifier of the target subscription node where the target subscription sub-node is located, and receives the target distribution message fed back by the subscription access node. The target subscription node is any one of the plurality of subscription nodes, and the target subscription sub-node is any one of the plurality of subscription sub-nodes under the target subscription node. The subscription access node is used to perform unit splitting on the multiple message distribution nodes based on multiple message transmission units, to obtain at least one message transmission unit to be distributed by each message distribution node; to send message acquisition requests to the multiple message distribution nodes simultaneously; and to integrate the target messages extracted by the multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier. Each message distribution node is configured to, in response to the message acquisition request, extract the target message of the message delivery unit corresponding to the target node identifier from at least one message delivery unit to be distributed by itself, and feed back the target message extracted by itself to the subscription access node.

9. The system according to claim 8, characterized in that, The system also includes: a message publisher, a message writer node, and a message storage node, wherein: The message publisher is used to send the published message to the message writing node; The message writing node is used to write the published message to the message storage node through micro-batch processing.

10. A message processing device, characterized in that, The device includes: The first unit splitting module is used to respond to message read requests sent by target subscription sub-nodes in the target message subscriber. Based on multiple message passing units, it splits multiple message distribution nodes into units to obtain at least one message passing unit to be distributed by each message distribution node. The message read request carries the target node identifier of the target subscription node where the target subscription sub-node is located. The target subscription node is any one of the multiple subscription nodes under the target message subscriber, and the target subscription sub-node is any one of the multiple subscription sub-nodes under the target subscription node. The multiple subscription nodes can read different messages, and the multiple subscription sub-nodes under the same subscription node can read the same messages. The first concurrent invocation module is used to concurrently invoke the multiple message distribution nodes to extract the target message of the message transmission unit corresponding to the target node identifier from at least one message transmission unit to be distributed by itself; The message integration module is used to integrate the target messages extracted by the multiple message distribution nodes to obtain the target distribution message corresponding to the target node identifier. The message feedback module is used to send the target distribution message back to the target subscription sub-node.

11. A message processing device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the message processing method as described in any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the message processing method as described in any one of claims 1 to 7.

13. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the message processing method as described in any one of claims 1 to 7.

Images