Message processing device and method
By designing a message processing device that encapsulates Redis Stream and multi-party underlying commands, the complex interaction problem of Redis Stream and multi-party parties is solved, the reliability and throughput of message processing are improved, and the interaction process is simplified.
Patent Information
- Application Number
- CN202411411961.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-17
AI Technical Summary
In the prior art, the underlying commands between Redis Stream and multiple parties are lacking in packaging, resulting in reduced reliability of message processing and complex interactions.
A message processing device is designed, including a sending module and a consumption module. By encapsulating Redis Stream and multiple underlying commands, the reliable transmission and processing of messages is realized, reducing interaction complexity.
Improve the reliability and throughput of message processing, simplify the interaction between Redis Stream and multiple parties, and enhance the stability of message processing.
Smart Images
Figure CN119322689A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of message processing, and in particular to a message processing device and method. Background Art
[0002] Since Redis Stream has a powerful message queue capability, Redis Stream is usually used in related technologies to implement message processing. However, in related technologies, there is a lack of encapsulation of many underlying commands between Redis Stream and multiple parties, which makes the interaction between Redis Stream and multiple parties complicated, thereby reducing the reliability of message processing. Therefore, how to improve the reliability of message processing is a key issue that technicians in this field are concerned about. Summary of the invention
[0003] Based on the above problems, the present application provides a message processing device and method to improve the reliability of message processing.
[0004] The embodiments of the present application disclose the following technical solutions:
[0005] The first aspect of the present application provides a message processing device. The message processing device includes a sending module and a consumption module, and the consumption module includes a message reading unit and a message processing unit:
[0006] The sending module is used to send the target message submitted by the target object on the client to the target RedisStream, so as to send the target message to the target consumer group in the target Redis Stream through the target Redis Stream, wherein the target Redis Stream has the capability of message queue;
[0007] The message reading unit is used to read the target message in the target consumption group and send the target message to the message processing unit;
[0008] The message processing unit is used to send the target message to the first consumer corresponding to the first consumer name in the target consumer group, so as to perform a message processing operation on the target message through the first consumer.
[0009] In an achievable implementation, the target consumption group includes a message to be processed table, and the target message exists in the message to be processed table;
[0010] The message processing unit is further configured to receive a processing completion instruction from the first consumer for the target message, and remove the target message from the message to be processed table based on the processing completion instruction.
[0011] In an achievable implementation, the consumption module further includes a consumer registration unit and a consumer grouping unit;
[0012] The consumer registration unit is used to register multiple consumers, and obtain consumer information corresponding to the multiple consumers based on the multiple consumers, and send the consumer information corresponding to the multiple consumers to the consumer grouping unit, wherein the consumer information of each consumer includes a consumer group name, a consumer name and a RedisStream target name.
[0013] In an achievable implementation, the consumer grouping unit is used to group the consumer names in the consumer information corresponding to the multiple consumers based on the consumer group names in the consumer information corresponding to the multiple consumers, to obtain multiple consumer groups, wherein each consumer group includes multiple different consumer names;
[0014] The consumer grouping unit is further configured to add the multiple consumer groups to a target Redis Stream based on the Redis Stream target name, wherein the target consumer group is any one of the multiple consumer groups.
[0015] In an achievable implementation, the target message includes the Redis Stream target name;
[0016] The sending module is specifically used to send the target message submitted by the target object on the client to the target Redis Stream corresponding to the Redis Stream target name based on the Redis Stream target name.
[0017] In an achievable implementation, the consumption module further includes a message transfer unit;
[0018] The message transfer unit is configured to transfer the target message to a second consumer corresponding to the second consumer name in the target consumer group if the first consumer does not process the target message and the duration of not processing the target message exceeds a first preset duration, so as to perform a message processing operation on the target message through the second consumer;
[0019] The second consumer name is any one of a plurality of different consumer names in the target consumer group except the first consumer name.
[0020] In one achievable implementation, the consumption module further includes a consumer clearing unit;
[0021] The consumer clearing unit is used to call a consumer acquisition command to obtain consumers corresponding to the multiple consumer groups, and clear consumers corresponding to the multiple consumer groups for which no messages exist and whose idle time exceeds a second preset time.
[0022] In an achievable implementation, the message processing device includes a compression module;
[0023] The compression module is used to compress the number of messages of the multiple messages in the target Redis Stream to obtain the number of compressed messages.
[0024] In an achievable implementation, the message reading unit is specifically configured to call a message reading command, obtain a target message in the target consumption group, and send the target message to the message processing unit.
[0025] The second aspect of the present application provides a message processing method. The message processing method is applied to the message processing device described in the first aspect, wherein the message processing device includes a sending module and a consumption module, the consumption module includes a message reading unit and a message processing unit, and the message processing method includes:
[0026] The target message submitted by the target object on the client is sent to the target RedisStream through the sending module, so as to send the target message to the target consumer group in the target Redis Stream through the target Redis Stream, wherein the target Redis Stream has the capability of message queue;
[0027] Reading the target message in the target consumption group through the message reading unit, and sending the target message to the message processing unit;
[0028] The target message is sent to the first consumer corresponding to the first consumer name in the target consumer group through the message processing unit, so that the target message is processed by the first consumer to obtain a processed message.
[0029] Compared with the prior art, this application has the following beneficial effects:
[0030] The message processing device in the present application includes a sending module and a consumption module. The sending module is used to send the target message submitted by the target object on the client to the target Redis Stream, so as to send the target message to the target consumption group in the target Redis Stream through the target Redis Stream, wherein the target Redis Stream has the ability of message queue; the message reading unit in the consumption module is used to read the target message in the target consumption group and send the target message to the message processing unit; the message processing unit in the consumption module is used to send the target message to the first consumer corresponding to the first consumer name in the target consumption group, so as to perform message processing operations on the target message through the first consumer.
[0031] It can be seen that in this application, the sending module can send the target message submitted by the client to the target RedisStream, the message reading unit can read the target message from the target consumer group in the target Redis Stream, and the message processing unit can send the target message to the consumer corresponding to the consumer name in the target consumer group. In this way, in this application, many underlying commands can be encapsulated in the sending module and the consumer module, so that the interaction between Redis Stream and multiple parties can be realized through the encapsulated sending module and consumer module, and the complexity of the interaction between Redis Stream and multiple parties can be reduced to a certain extent, thereby improving the reliability of message processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 A schematic diagram of a message processing device provided in an embodiment of the present application;
[0034] Figure 2 A schematic diagram of a consumption module in a message processing device provided in an embodiment of the present application;
[0035] Figure 3 A flowchart of a message processing device in a message processing device provided in an embodiment of the present application;
[0036] Figure 4 A flowchart of a message processing method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0037] As described above, since Redis Stream has a powerful message queue capability, Redis Stream is usually used in related technologies to implement message processing. However, in related technologies, there is a lack of encapsulation of many underlying commands between Redis Stream and multiple parties, which makes the interaction between Redis Stream and multiple parties complicated. That is, in related technologies, there is a lack of encapsulation of many commands between Redis Stream and the sender, and a lack of encapsulation of many commands between Redis Stream and the consumer, which leads to confusion of commands between Redis Stream and the sender and the consumer, making the interaction between Redis Stream and the sender and the consumer complicated, thereby reducing the reliability of message processing. Therefore, how to improve the reliability of message processing is a key issue that technicians in this field are concerned about.
[0038] In view of the above problems, a solution is provided in an embodiment of the present application, and a message processing device and method are proposed, aiming to improve the reliability of message processing. The message processing device in the technical solution of the present application includes a sending module and a consumption module, wherein the consumption module includes a message reading unit and a message processing unit. The sending module is used to send the target message submitted by the target object on the client to the target Redis Stream, so as to send the target message to the target consumer group in the target Redis Stream through the target Redis Stream, wherein the target Redis Stream has the ability of message queue; the message reading unit is used to read the target message in the target consumer group and send the target message to the message processing unit; the message processing unit is used to send the target message to the first consumer corresponding to the first consumer name in the target consumer group, so as to perform message processing operations on the target message through the first consumer.
[0039] It can be seen that in this application, the sending module can send the target message submitted by the client to the target RedisStream, the message reading unit in the consumption module can read the target message from the target consumption group in the target Redis Stream, and the message processing unit in the consumption module can send the target message to the consumer corresponding to the consumer name in the target consumption group. In this way, in this application, many underlying commands can be encapsulated in the sending module and the consumption module, so that the interaction between Redis Stream and multiple parties can be realized through the encapsulated sending module and consumption module, and the complexity of the interaction between Redis Stream and multiple parties can be reduced to a certain extent, thereby improving the reliability of message processing.
[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0041] Next, the technical terms that may appear in this application are explained.
[0042] Redis Stream: A data structure that has the capability of a message queue.
[0043] See also Figure 1 , which is a schematic diagram of a message processing device provided in an embodiment of the present application. Figure 1 As shown:
[0044] exist Figure 1 The message processing device 100 includes a sending module 101 and a consumption module 102, wherein the consumption module 102 includes a message reading unit 21 and a message processing unit 22. Specifically, the sending module 101 can be used to send the target message submitted by the target object on the client to the target Redis Stream, wherein the target message can be the target question entered by the target object in the "Search" bar of the client, and the target message can also be the target event generated by the target object clicking the "Download" control on the client. It should be noted that the target message is not limited in this application and can also be determined in actual applications.
[0045] After the sending module 101 sends the target message submitted by the target object on the client to the target Redis Stream, the target message can be sent to the target consumer group in the target Redis Stream through the target Redis Stream, so that the message reading unit 21 in the consumer module 102 can read the target message from the target consumer group in the subsequent process. The target Redis Stream has the ability of a message queue, that is, the target Redis Stream can store the target message in the memory of the target Redis Stream, so that the subsequent reading of the target message can be supported. In this way, the sending module 101 in the present application encapsulates many commands of the Redis Stream and the client, and can better implement the interaction between the Redis Stream and the client through the sending module 101.
[0046] Further, the message reading unit 21 in the consumption module 102 can be used to read the target message from the target consumption group of the target Redis Stream, that is, the message reading unit 21 can read new messages from the Redis Stream, so as to ensure that new messages are read in real time and avoid the loss of message reading. After that, the message reading unit 21 can send the target message to the message processing unit 22, so that the message processing unit 22 can process the target message in the subsequent process.
[0047] Furthermore, the message processing unit 22 in the consumption module 102 can be used to send the target message to the first consumer corresponding to the first consumer name in the target consumption group, so as to perform a message processing operation on the target message through the first consumer. The consumption module 102 can determine the consumer corresponding to the consumer name according to the consumer name in the target consumption group to realize the sending of the target message, and the consumer is located in the consumption end. In this way, the consumption module 102 in the present application encapsulates many commands of Redis Stream and the consumption end, and can better realize the interaction between Redis Stream and the consumption end through the consumption module 102.
[0048] Next, the sending module 101 and the consuming module 102 in this application are introduced one by one. The introduction of the sending module 101 is as follows:
[0049] In an achievable implementation, the target message includes a Redis Stream target name, wherein the Redis Stream target name is used to indicate to which Redis Stream the target message needs to be sent. It is understandable that, since the client may generate many messages at the same time, multiple Redis Streams may exist simultaneously in the present application to support message processing. At this time, the sending module 101 can determine that the target message needs to be sent to the target Redis Stream according to the Redis Stream target name, and can send the target message submitted by the target object on the client to the target Redis Stream corresponding to the Redis Stream target name, so as to ensure that the message can be processed normally in the subsequent process.
[0050] The sending module 101 can call the Redis XADD command to add the target message to the target Redis Stream corresponding to the Redis Stream target name. The sample code of the Redis XADD command is as follows: Redis XADD command: XADD stream name * payload_class com.neu.bean.MyMessage payload_json{"name":"luo","age":18}. The Redis XADD command is not specifically limited in this application.
[0051] It should be noted that the target Redis Stream can automatically generate a message name for the target message to facilitate subsequent search for the target message. The target Redis Stream can also convert the target message into a converted target message, which can be stored in the memory of the target Redis Stream. For example: if the target message can be payload_class, after conversion, the converted target message can be the full path of the class name of the target message; if the target message can be payload_json, after conversion, the converted target message can be the JSON serialized string of the target message.
[0052] like Figure 2 As shown, Figure 2 A schematic diagram of a consumption module in a message processing device provided in an embodiment of the present application. Figure 2 The consumption module 102 is specifically introduced. Figure 2 The middle consumption module 102 includes a message reading unit 21 , a message processing unit 22 , a consumer registration unit 23 , a consumer grouping unit 24 , a message transfer unit 25 and a consumer clearing unit 26 .
[0053] In an achievable implementation, the consumer registration unit 23 in the consumption module 102 of the present application can be used to register multiple consumers in different consumer terminals, so that when there are many messages to be processed later, consumers at different consumer terminals can be called to implement message processing to ensure message processing efficiency. And the consumer registration unit 23 can be used to obtain consumer information corresponding to multiple consumers according to multiple consumers, wherein the consumer information of each consumer includes a consumer group name, a consumer name, and a Redis Stream target name. After that, the consumer information corresponding to the multiple consumers can be sent to the consumer grouping unit 24, so that the consumer grouping unit 24 can group the multiple consumers based on the consumer group name in the subsequent process.
[0054] It can be understood that the consumer group name in the consumer information of the consumer can indicate the message under which consumer group the consumer needs to process; the consumer name in the consumer information of the consumer can exist in the consumer group of Redis Stream, so that the consumer can be called based on the consumer name under the consumer group in the future; the RedisStream target name is used to indicate the message under which Redis Stream the consumer needs to process. And the consumer information of each consumer also includes the message processor corresponding to the consumer, so that the subsequent message processing unit 22 can send the target message to the corresponding message processor to realize message processing.
[0055] In an achievable implementation, the consumer grouping unit 24 can be used to group the consumer names in the consumer information corresponding to the multiple consumers based on the consumer group names in the consumer information corresponding to the multiple consumers, so as to obtain multiple consumer groups, wherein each consumer group includes multiple different consumer names. For example, the multiple consumers include consumer A, consumer B, and consumer C, the consumer name of consumer A is A, the consumer name of consumer B is B, and the consumer name of consumer C is C, the consumer group name of consumer A is group 1, the consumer group name of consumer B is group 2, and the consumer group name of consumer C is group 1.
[0056] At this time, according to the consumer group name of consumer A and the consumer group name of consumer C, the consumer name of consumer A (A) and the consumer name of consumer C (C) can be grouped into a first consumer group; and according to the consumer group name of consumer B, the consumer name of consumer B (B) can be grouped into a second consumer group, wherein the first consumer group and the second consumer group can respectively manage different types of messages. In this way, the mutual impact between different services can be reduced.
[0057] And because multiple Redis Streams can exist simultaneously in the present application to support the processing of many messages, the consumer grouping unit 24 in the present application can also be used to add multiple consumer groups to the target Redis Stream based on the Redis Stream target name, where the target consumer group is any one of the multiple consumer groups. It can be understood that the target Redis Stream can send the target message to the target consumer group in the target Redis Stream. If there are many pending messages in the target consumer group (i.e., no new messages can be received), the target Redis Stream can send the target message to any consumer group in the target Redis Stream except the target consumer group.
[0058] It should also be noted that after the consumer grouping unit 24 in this application obtains the consumer information corresponding to multiple consumers, it can first determine whether the consumer group corresponding to the consumer group name in the Redis Stream already exists according to the consumer group name in the consumer information corresponding to multiple consumers. If the consumer group corresponding to the consumer group name in the Redis Stream already exists, the consumer name can be directly added to this consumer group; if the consumer group corresponding to the consumer group name in the Redis Stream does not exist, the Redis XGROUP CREATE command can be called to create a consumer group, and then the consumer name can be added to this consumer group.
[0059] Among them, the example code of the Redis XGROUP CREATE command is as follows. The Redis XGROUP CREATE command: XGROUP CREATE stream name group name $, and this application does not make specific limitations on the Redis XGROUP CREATE command. In this way, this application can perform consumer registration operations and consumer grouping operations through the consumer registration unit and the consumer grouping unit in advance, so that consumers can be directly called in the subsequent process, which can improve the message processing efficiency to a certain extent.
[0060] In an implementable embodiment, the message reading unit 21 in the consumption module 102 in this application is specifically used to call a message reading command to read and process the target Redis Stream to obtain the target message in the target consumer group of the target Redis Stream, and send the target message to the message processing unit 22, where the message reading command includes the Redis XGROUP READ command. The example code of the Redis XGROUP READ command is as follows. The Redis XGROUP READ command: XREADGROUP GROUP <group name consumer name BLOCK blocking duration COUNT maximum number of reads ETREAM stream name>, and this application does not make specific limitations on the Redis XGROUP READ command.
[0061] It can be understood that since there may be multiple consumers on each consumer side, in this application, an independent thread can be started for each consumer side, and the Redis XGROUP READ command can be called within this thread to read the new messages in the target Redis Stream, that is, the messages that have not been delivered to any consumer. In this way, the efficiency and throughput of message processing can be ensured.
[0062] In an achievable implementation, the target consumer group includes a message to-be-processed table, that is, each consumer group has a message to-be-processed table for each consumer group, which is convenient for message management. It is understandable that when the message reading unit 21 sends the target message to the message processing unit 22, the message processing unit 22 can record the message name corresponding to the target message in the message to-be-processed table, and at the same time record the sending time and number of times the target message is sent to the message processing unit 22.
[0063] After that, the message processing unit 22 can send the target message to the first consumer corresponding to the first consumer name in the target consumer group in an independent thread, so as to perform a message processing operation on the target message through the first consumer. It should be noted that in this application, the target message can be randomly sent to any consumer group in the target Redis Stream and any consumer under the consumer group. And at this time, the consumer name of the first consumer can be recorded in the message to be processed table to facilitate the management of messages during the message processing process.
[0064] Table 1
[0065] Message Name Send time Number of times sent Consumer Name XX 2024-06-01,00:00 1 A
[0066] As shown in Table 1, Table 1 is a schematic diagram of a message to-be-processed table in a message processing device provided in an embodiment of the present application. In Table 1, the message name of the target message may be XX, the sending time of the target message may be 2024-06-01, 00:00, the number of times the target message is sent may be 1, and the consumer name of the target message may be A. It should be noted that the consumer name in the message to-be-processed table is the consumer name corresponding to the consumer who has not returned the processing completion instruction of the target message.
[0067] Specifically, the message processing unit 22 can call the message processor corresponding to the first consumer corresponding to the first consumer name in the target consumer group in an independent thread to process the target message through the message processor corresponding to the first consumer. And the message processing unit 32 can also call the Redis XACK command to receive the processing completion instruction for the target message returned by the message processor corresponding to the first consumer. The sample code of the Redis XACK command is as follows: RedisXACK command: XACK stream name group name message ID. The Redis XACK command is not specifically limited in this application.
[0068] If the processing completion instruction indicates that the message processor has completed processing of the target message and no processing exception has occurred, the target message in the message to be processed table can be removed according to the processing completion instruction, so that the message can be prevented from being processed by all message processors again. If the processing completion instruction indicates that the message processor has not completed processing of the target message or a processing exception has occurred (such as a consumer end service downtime, network anomaly, etc. causing message processing exception), the target message is recorded in the log so that the subsequent message transfer unit 25 can transfer the target message. In this way, the message can be prevented from being processed by the same message processor again.
[0069] It should also be noted that, since the target Redis Stream has converted the target message into the converted target message, the message processing unit 22 can also be used to convert the converted target message into the target message, so as to facilitate the subsequent processing of the target message by the consumer, such as converting the full path of the class name of the target message (the converted target message) into payload_class (the target message), etc.
[0070] In a feasible implementation, the message transfer unit 25 in the consumption module 102 can be used to determine whether the first consumer has not processed the target message, and whether the time for which the first consumer has not processed the target message exceeds a first preset time, where the first preset time can be 2 minutes, or the first preset time can be 3 minutes, which is not specifically limited here.
[0071] If the first consumer does not process the target message, and the time period during which the first consumer does not process the target message exceeds the first preset time period, the message transfer unit 25 can transfer the target message to the second consumer corresponding to the second consumer name in the target consumer group, so that the second consumer can perform a message processing operation on the target message; otherwise, the message transfer operation is not performed, wherein the second consumer name is any one of the multiple different consumer names in the target consumer group except the first consumer name. In this way, it is possible to avoid the message from being ignored for a long time to a certain extent, and to ensure the effective processing of the message, and to successfully process the message in the case of message abnormality.
[0072] Specifically, in the present application, the Redis XAUTOCLAIM command can be called in a separate thread to transfer the target message to the second consumer corresponding to the second consumer name in the target consumer group. Since the Redis XAUTOCLAIM command can ensure that when multiple consumers transfer the ownership of the same message at the same time, only one client will successfully transfer the ownership. This can ensure that the target message will not be transferred to multiple consumers at the same time.
[0073] Among them, the sample code of the Redis XAUTOCLAIM command is as follows: Redis XAUTOCLAIM command: XAUTOCLAIM stream name group name consumer name idle time 0-0 COUNT transfer quantity. The Redis XAUTOCLAIM command is not specifically limited in this application.
[0074] In an achievable implementation, the consumer clearing unit 26 in the consumption module 102 can be used to call a consumer acquisition command to obtain consumers (i.e., all consumers) corresponding to multiple consumption groups. It can also be determined whether there are consumers for which there is no message corresponding to the consumers of the multiple consumption groups, and whether the idle time of the consumers for which there is no message exceeds a second preset time, wherein the second preset time can be 10 minutes, or 20 minutes, which is not specifically limited here.
[0075] If there are consumers for which there are no messages among the consumers corresponding to the multiple consumer groups, and the idle time of the consumers for which there are no messages exceeds the second preset time, the consumer clearing unit 26 clears the consumers for which there are no messages and whose idle time exceeds the second preset time; otherwise, the consumer clearing operation is not performed. Specifically, in the present application, the consumer clearing unit 26 can call the Redis XINFO CONSUMERS command in an independent thread to obtain the consumers corresponding to the multiple consumer groups, and clear the consumers for which there are no messages and whose idle time exceeds the second preset time. This can optimize the memory to a certain extent.
[0076] Among them, the sample code of the Redis XINFO CONSUMERS command is as follows: Redis XINFO CONSUMERS command: XINFO CONSUMERS stream name group name. The Redis XINFO CONSUMERS command is not specifically limited in this application.
[0077] In addition, the message processing device 100 in the present application also includes a compression module, which can be used to compress the number of messages of multiple messages in the target Redis Stream within a preset time period to obtain the compressed number of messages, wherein the preset time period can be 00:00-00:10 every day, and the preset time period can also be 00:00-00:15 every day, which is not specifically limited here. Specifically, in the present application, the Redis XTRIM command can be called in an independent thread to compress the number of messages of multiple messages in the target RedisStream to a preset number of messages, so that the number of messages in the message queue is within a reasonable range. In this way, a large backlog of messages in the message queue can be avoided. Among them, the example code of the Redis XTRIM command is as follows, Redis XTRIM command: XTRIM stream name XTRIM number of messages, and the RedisXTRIM command is not specifically limited in the present application.
[0078] like Figure 3 As shown, Figure 3 A flowchart of a message processing device in a message processing device provided in an embodiment of the present application. Figure 3 In the example, the sending module 101 can be used to send the target message submitted by the target object on the client to the target RedisStream, and the target Redis Stream can send the target message to the target consumption group; the compression module 103 can be used to compress the number of messages in the target Redis Stream; the consumer registration unit 23 in the consumption module 102 can be used to register multiple consumers in different consumption ends; the consumer grouping unit 24 in the consumption module 102 can be used to group multiple consumers, so that the consumer names corresponding to the multiple consumers can exist in multiple consumption groups of the target Redis Stream respectively.
[0079] The message reading unit 21 in the consumption module 102 can be used to read the target message in the target consumption group in the target Redis Stream; the message processing unit 22 in the consumption module 102 can be used to send the target message to the first consumer corresponding to the first consumer name in the target consumption group, so as to perform a message processing operation on the target message through the first consumer; the message transfer unit 25 in the consumption module 102 can be used to transfer the target message to the second consumer corresponding to the second consumer name in the target consumption group, so as to perform a message processing operation on the target message through the second consumer; the consumer clearing unit 26 in the consumption module 102 can be used to clear consumers for which there is no message and whose idle time exceeds the second preset time. In this way, the present application realizes the encapsulation of many underlying commands, reduces the complexity of the interaction between Redis Stream and multiple parties, and improves the reliability of message processing.
[0080] It can be seen that in this optional solution, the sending module can send the target message submitted by the client to the target RedisStream, the message reading unit in the consumption module can read the target message from the target consumption group in the target Redis Stream, and the message processing unit in the consumption module can send the target message to the consumer corresponding to the consumer name in the target consumption group. In this way, in this application, many underlying commands can be encapsulated in the sending module and the consumption module, so that the interaction between Redis Stream and multiple parties can be realized through the encapsulated sending module and consumption module, and the complexity of the interaction between Redis Stream and multiple parties can be reduced to a certain extent, thereby improving the reliability of message processing and improving the throughput of message processing.
[0081] A message processing method provided by the present application is described below through another embodiment.
[0082] See also Figure 4 , which is a flow chart of a message processing method provided by an embodiment of the present application, such as Figure 4 As shown, the method is applied to the above-mentioned message processing device, the message processing device includes a sending module and a consumption module, wherein the consumption module includes a message reading unit and a message processing unit, and the message processing method may include:
[0083] S401: Sending a target message submitted by a target object on a client to a target RedisStream through the sending module, so as to send the target message to a target consumer group in the target Redis Stream through the target Redis Stream, wherein the target Redis Stream has a message queue capability.
[0084] It should be noted that the message sending process of the sending module in step S401 is similar to the message sending process of the sending module in the above-mentioned message processing device, and will not be repeated here.
[0085] S402: Read the target message in the target consumption group through the message reading unit, and send the target message to the message processing unit.
[0086] It should be noted that the message reading process of the message reading unit in the consumption module in step S402 is similar to the message reading process of the message reading unit in the consumption module in the above-mentioned message processing device, and will not be repeated here.
[0087] S403: Sending the target message to the first consumer corresponding to the first consumer name in the target consumer group through the message processing unit, so that the target message is processed by the first consumer to obtain a processed message.
[0088] It should be noted that the message processing process of the message processing unit in the consumption module in step S403 is similar to the message processing process of the message processing unit in the consumption module in the above-mentioned message processing device, and will not be repeated here.
[0089] In summary, in this embodiment, the sending module and the consuming module encapsulated in the message processing device can realize the processing of messages, and in actual applications, the client only needs to call the sending module and the consuming module through a simple interface to complete the sending and consumption processing of messages, thereby improving the reliability of message processing and reducing the complexity of the interaction between Redis Stream and multiple parties.
[0090] In addition, the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0091] It should be noted that the "first" and "second" (if any) in the names such as "first" and "second" mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.
[0092] The above is a detailed introduction to a message processing device and method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.
Claims
1. A message processing device, characterized in that: The message processing device comprises a sending module and a consumption module, and the consumption module comprises a message reading unit and a message processing unit; The sending module is used to send the target message submitted by the target object on the client to the target Redis Stream, so as to send the target message to the target consumer group in the target Redis Stream through the target Redis Stream, wherein the target Redis Stream has the capability of message queue; The message reading unit is used to read the target message in the target consumption group and send the target message to the message processing unit; The message processing unit is used to send the target message to the first consumer corresponding to the first consumer name in the target consumer group, so as to perform a message processing operation on the target message through the first consumer.
2. The device according to claim 1, characterized in that The target consumption group includes a message to be processed table, and the target message exists in the message to be processed table; The message processing unit is further configured to receive a processing completion instruction from the first consumer for the target message, and remove the target message from the message to be processed table based on the processing completion instruction.
3. The device according to claim 1, characterized in that The consumption module also includes a consumer registration unit and a consumer grouping unit; The consumer registration unit is used to register multiple consumers, and obtain consumer information corresponding to the multiple consumers based on the multiple consumers, and send the consumer information corresponding to the multiple consumers to the consumer grouping unit, wherein the consumer information of each consumer includes a consumer group name, a consumer name and a RedisStream target name.
4. The device according to claim 3, characterized in that The consumer grouping unit is used to group the consumer names in the consumer information corresponding to the multiple consumers based on the consumer group names in the consumer information corresponding to the multiple consumers, so as to obtain multiple consumer groups, wherein each consumer group includes multiple different consumer names; The consumer grouping unit is further configured to add the multiple consumer groups to a target Redis Stream based on the Redis Stream target name, wherein the target consumer group is any one of the multiple consumer groups.
5. The device according to claim 3, characterized in that The target message includes the Redis Stream target name; The sending module is specifically used to send the target message submitted by the target object on the client to the target Redis Stream corresponding to the Redis Stream target name based on the Redis Stream target name.
6. The device according to claim 1, characterized in that The consumption module also includes a message transfer unit; The message transfer unit is configured to transfer the target message to a second consumer corresponding to the second consumer name in the target consumer group if the first consumer does not process the target message and the duration of not processing the target message exceeds a first preset duration, so as to perform a message processing operation on the target message through the second consumer; The second consumer name is any one of a plurality of different consumer names in the target consumer group except the first consumer name.
7. The device according to claim 1, characterized in that The consumption module also includes a consumer removal unit; The consumer clearing unit is used to call a consumer acquisition command to obtain consumers corresponding to the multiple consumer groups, and clear consumers corresponding to the multiple consumer groups for which no messages exist and whose idle time exceeds a second preset time.
8. The device according to claim 1, characterized in that The message processing device includes a compression module; The compression module is used to compress the number of messages of the multiple messages in the target Redis Stream to obtain the number of compressed messages.
9. The device according to claim 1, characterized in that The message reading unit is specifically used to call a message reading command, obtain the target message in the target consumption group, and send the target message to the message processing unit.
10. A message processing method, characterized in that: The message processing method is applied to the message processing device according to any one of claims 1 to 9, wherein the message processing device comprises a sending module and a consumption module, the consumption module comprises a message reading unit and a message processing unit, and the message processing method comprises: The target message submitted by the target object on the client is sent to the target Redis Stream through the sending module, so as to send the target message to the target consumer group in the target Redis Stream through the target Redis Stream, wherein the target Redis Stream has the capability of message queue; Reading the target message in the target consumption group through the message reading unit, and sending the target message to the message processing unit; The target message is sent to the first consumer corresponding to the first consumer name in the target consumer group through the message processing unit, so that the target message is processed by the first consumer to obtain a processed message.