Data processing method, system and server based on message queue

Abstract

The embodiment of the present invention provides a data processing method, system and server based on a message queue, which belongs to the field of communication technology. The method includes parsing the target key field and the target message sequence number of the target message obtained from the message queue, and obtaining the maximum message sequence number corresponding to the target key field from the cache server; judging whether to consume the target message according to the size relationship between the maximum message sequence number and the target message sequence number, and whether the target message processing mark is stored in the cache server; when the target message is consumed successfully, the maximum message sequence number stored in the cache server is updated with the target message sequence number. The present invention can ensure that the same message is consumed only once under the premise of ensuring data consistency.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a data processing method, system and server based on a message queue. Background Art

[0002] When synchronous communication is used between business components, the interaction efficiency is low. For example, after the requester sends a business request to the server, it needs to wait synchronously for the server to return a business response, wasting idle computing resources. In addition, when each requester communicates directly with the server, when there is a large amount of concurrent traffic, the server's resource scheduling is frequent, resulting in low effective resource utilization on the server, which in turn leads to low processing efficiency and slow response speed on the server.

[0003] In order to ensure high availability and high performance of the business system, the synchronous communication mode between the requester and the server can be changed to an asynchronous communication mode. In addition, in order to alleviate the processing pressure of the server when there is a large amount of concurrent traffic, a message queue (MQ) can be added between the requester and the server so that the server can obtain business requests from the message queue in batches to reduce the peak of concurrent traffic.

[0004] The inventor of this patent application discovered through studying the communication process between the request end, the message queue and the server end that the prior art has at least the following problems: adding a message queue between the request end (which may be called the producer) and the server end (which may be called the consumer) reduces the reliability of message transmission. In order to overcome the situation where anomalies occur during the transmission of the message from the producer to the message queue, and anomalies occur at the message queue, the producer can repeatedly send the same message to the message queue. Similarly, in order to overcome the situation where anomalies occur during the transmission of the message from the message queue to the consumer, and anomalies occur at the consumer, the message queue can repeatedly provide the same message to the consumer. This may cause the same message to be consumed repeatedly and cause data inconsistency problems after the consumer processes the invalid message. Summary of the invention

[0005] The purpose of this application is to provide a data processing method, system and server based on a message queue to solve some or all of the problems existing in the prior art. The technical solution is as follows:

[0006] In a first aspect, a data processing method based on a message queue is provided, which is applied to a consumer end, and the method includes:

[0007] Parsing a target key field and a target message sequence number of a target message obtained from a message queue, and obtaining a maximum message sequence number corresponding to the target key field from a cache server; wherein the target message sequence number is an identification code with a time sequence;

[0008] Determine whether to consume the target message according to the size relationship between the maximum message sequence number and the target message sequence number and whether the target message processing flag is stored in the cache server;

[0009] When the target message is consumed successfully, the maximum message sequence number stored in the cache server is updated with the target message sequence number.

[0010] Optionally, judging whether to consume the target message according to a size relationship between the maximum message sequence number and the target message sequence number and whether a target message processing flag is stored in the cache server includes:

[0011] If the maximum message sequence number is less than the target message sequence number and the target message being processed does not exist in the cache server, the target message being processed is recorded in the cache server and the target message is consumed;

[0012] If the maximum message sequence number is not less than the target message sequence number, or the maximum message sequence number is less than the target message sequence number and there is a target message processing flag in the cache server, the target message is ignored.

[0013] In implementation, each message obtained by the consumer carries an identification code with a time sequence. After each successful consumption of a message, the consumer can record the message sequence number of the message in the cache server. Before consuming the target message, the consumer can compare the message sequence number of the target message with the maximum message sequence number recorded in the cache server. If the message sequence number of the target message is larger, it means that the target message is the latest message and has not been successfully consumed; if the message sequence number of the target message is smaller, it means that the target message is not the latest message; if the two are equal, it means that the target message is the latest message, but has been successfully consumed. Based on the message sequence number of each message, the consumer can determine to consume the latest message that has not been successfully consumed, and ignore invalid messages or messages that have been consumed. In this way, it can be ensured that only the latest message is consumed, and the same message is only consumed once.

[0014] Optionally, before parsing the target key field and the target message sequence number of the target message obtained from the message queue, the method further includes:

[0015] Obtain a target batch of messages from a message queue, and query the cache server whether there is a target batch being processed mark; wherein the target batch of messages includes the target message;

[0016] If it exists, the target batch message is discarded; if it does not exist, the target batch processing mark is recorded in the cache server.

[0017] In implementation, the consumer can obtain messages in batches from the message queue. Before consuming each batch of messages, the consumer can record the processing mark corresponding to this batch of messages in the cache server. After obtaining the target batch of messages, the consumer can query whether the target batch processing mark corresponding to the target batch of messages is recorded in the cache server. If it does not exist, it means that the target batch message may not have been obtained by any consumer; if it exists, it means that the target batch message has been obtained by any consumer and may have been consumed. Based on the target batch processing mark, the consumer can discard the entire batch of messages after determining that the target batch message has been obtained by any consumer. In this way, there is no need to process the messages in the target batch message one by one, saving computing resources on the consumer side.

[0018] Optionally, obtaining the target batch of messages from the message queue includes pulling the target batch of messages from the message queue or receiving the target batch of messages pushed by the message queue;

[0019] After pulling the target batch of messages from the message queue, the method further includes:

[0020] Recording the maximum index value of the target batch of messages in the cache server, so as to pull the next batch of messages according to the maximum index value;

[0021] After receiving the target batch of messages pushed by the message queue, the method further includes:

[0022] A successful reception response is replied to the message queue, and the message queue records the maximum index value of the target batch of messages in the cache server, so that the message queue pushes the next batch of messages according to the maximum index value.

[0023] In implementation, the consumer can actively pull the target batch of messages from the message queue, or the message queue can actively push the target batch of messages to the consumer. After each successful pull of a batch of messages by the consumer, the maximum index of the pulled messages can be recorded in the cache server; after each successful push of a batch of messages by the message queue, the maximum index of the pushed messages can be recorded in the cache server. The maximum index is used to determine where to start pulling or pushing the next batch of messages.

[0024] Optionally, the method further includes:

[0025] When the target message is consumed successfully, the target message processing mark is deleted;

[0026] If the target message processing mark is stored in the cache server for a time period exceeding a first preset time period, an alarm is triggered.

[0027] In implementation, each time a message is successfully consumed by the consumer, the processing flag corresponding to the message stored in the cache server can be deleted. If the target message processing flag is stored in the cache server for too long, it means that the consumer is abnormal in processing the target message. At this time, triggering an alarm is helpful for troubleshooting.

[0028] Optionally, the method further includes:

[0029] When the target message consumption fails, the target message is moved to the abnormal message queue;

[0030] If there is any message in the abnormal message queue, an alarm is triggered.

[0031] In implementation, if the target message itself is abnormal, the consumer will not be able to successfully consume the target message. When the consumer fails to consume the target message, it can actively move the target message to the abnormal message queue to trigger an alarm, which is conducive to troubleshooting.

[0032] Optionally, ignoring the target message includes:

[0033] The target message is stored in a waiting queue, so that the consumer end obtains the target message from the waiting queue and determines again whether to consume the target message.

[0034] In implementation, the obtained duplicate messages are temporarily stored in a waiting queue, so that when data inconsistency occurs on the consumer side, valid messages can be re-obtained and consumed from the waiting queue, thereby ensuring data consistency.

[0035] Optionally, the method further includes:

[0036] When all messages in the target batch of messages are processed, the target batch processing mark is deleted;

[0037] If the storage time of the mark bit in the target batch processing in the cache server exceeds the second preset time length, an alarm is triggered.

[0038] In implementation, each time a batch of messages is successfully consumed by the consumer, the processing mark corresponding to the batch of messages stored in the cache server can be deleted. If the target batch of processing marks are stored in the cache server for too long, it means that the consumer has an abnormality in the process of processing the target batch of messages. At this time, triggering an alarm is helpful for troubleshooting.

[0039] In a second aspect, a data processing method based on a message queue is provided, which is applied to a message queue, and the method includes:

[0040] Query the maximum subscript value of the successfully pushed messages, and push the target batch of messages to the consumer end according to the maximum subscript value;

[0041] If a successful reception response is received from the consumer end, or if it is determined that the target batch is being processed in the cache server, it is determined that the target batch message is pushed successfully, and the maximum subscript value of the target batch message is recorded;

[0042] If the successful reception response is not received within the third preset time period, and the target batch processing mark does not exist in the cache server, it is determined that the target batch message push fails and an alarm is triggered;

[0043] The target batch processing mark is recorded in the cache server by the consumer end when the consumer end receives the target batch message.

[0044] During implementation, each time the consumer successfully obtains a batch of messages, it can reply to the message queue with a successful reception response corresponding to this batch of messages. And before starting to consume this batch of messages, the consumer can record the processing mark corresponding to this batch of messages in the cache server. After the message queue pushes the target batch of messages to the consumer, if it does not receive the successful reception response from the consumer in time, it means that the network transmission between the message queue and the consumer may be abnormal or the consumer may be abnormal. At this time, the message queue can determine whether the target batch of messages has not been obtained by any consumer by judging whether there is a target batch processing mark in the cache server. If it exists, it means that the target batch of messages has been obtained by any consumer and may have been consumed. At this time, the target batch of messages can be not pushed to avoid repeated push. If it does not exist, it means that the target batch of messages may not have been successfully obtained by any consumer. At this time, an alarm can be triggered to facilitate troubleshooting.

[0045] Optionally, the method further includes:

[0046] Each message in the target batch of messages carries a message sequence number, and the message sequence number of each message is an identification code with a time sequence;

[0047] The message sequence number is used to enable the consumer to determine whether to consume the message corresponding to the message sequence number.

[0048] In implementation, the producer can add a time-sequenced identification code to the message body of the message when generating the message. In this way, each message in the message queue carries a time-sequenced message sequence number, which can provide a basis for the consumer to determine whether the message is invalid or duplicated.

[0049] In a third aspect, a data processing system based on a message queue is provided, the system comprising a message queue and a consumer end, wherein the consumer end is used to:

[0050] Parsing a target key field and a target message sequence number of a target message obtained from a message queue, and obtaining a maximum message sequence number corresponding to the target key field from a cache server; wherein the target message sequence number is an identification code with a time sequence;

[0051] If the maximum message sequence number is not less than the target message sequence number, or the maximum message sequence number is less than the target message sequence number and there is a target message processing flag in the cache server, then the target message is ignored;

[0052] If the maximum message sequence number is less than the target message sequence number and the target message being processed does not exist in the cache server, the target message being processed is recorded in the cache server and the target message is consumed;

[0053] When the target message is consumed successfully, the maximum message sequence number stored in the cache server is updated with the target message sequence number.

[0054] Optionally, before parsing the target key field and the target message sequence number of the target message obtained from the message queue, the consumer end is further used to:

[0055] Obtain a target batch of messages from a message queue, and query the cache server whether there is a target batch being processed mark; wherein the target batch of messages includes the target message;

[0056] If it exists, the target batch message is discarded; if it does not exist, the target batch processing mark is recorded in the cache server.

[0057] Optionally, the consumer end pulls the target batch of messages from the message queue or receives the target batch of messages pushed by the message queue; wherein,

[0058] After pulling the target batch of messages from the message queue, the consumer end is further used to:

[0059] Recording the maximum index value of the target batch of messages in the cache server, so as to pull the next batch of messages according to the maximum index value;

[0060] After receiving the target batch message pushed by the message queue, the consumer end is further used to:

[0061] A successful reception response is replied to the message queue, and the message queue records the maximum index value of the target batch of messages in the cache server, so that the message queue pushes the next batch of messages according to the maximum index value.

[0062] Optionally, the consumer end is further used for:

[0063] When the target message is consumed successfully, the target message processing mark is deleted;

[0064] If the target message processing mark is stored in the cache server for a time period exceeding a first preset time period, an alarm is triggered.

[0065] Optionally, the consumer end is further used for:

[0066] When the target message consumption fails, the target message is moved to the abnormal message queue;

[0067] If there is any message in the abnormal message queue, an alarm is triggered.

[0068] Optionally, the consumer end is further used for:

[0069] When all messages in the target batch of messages are processed, the target batch processing mark is deleted;

[0070] If the storage time of the mark bit in the target batch processing in the cache server exceeds the second preset time length, an alarm is triggered.

[0071] Optionally, the message queue is used for:

[0072] Querying the maximum subscript value recorded in the cache server, and pushing a target batch message to the consumer end according to the maximum subscript value recorded in the cache server;

[0073] If the successful reception response is received, or it is determined that the target batch is being processed in the cache server, it is determined that the target batch message is pushed successfully, and the maximum subscript value of the target batch message is recorded in the cache server;

[0074] If the successful reception response is not received within the third preset time period, and the target batch processing mark does not exist in the cache server, it is determined that the target batch message push fails and an alarm is triggered.

[0075] In a fourth aspect, a server is provided, characterized in that the server includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the method described in the first aspect or the second aspect.

[0076] In a fifth aspect, a cache server is provided, characterized in that the cache server, a message queue and a consumer end jointly implement the method described in the first aspect or the second aspect.

[0077] In the sixth aspect, a computer-readable storage medium is provided, characterized in that at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method described in the first aspect or the second aspect.

[0078] As can be seen from the above, the beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least the following: First, the consumer end can determine whether the target message is an invalid message or a duplicate message by comparing the target message sequence number with the maximum message sequence number stored in the cache server; by querying whether there is a target message processing mark in the cache server, it can be further determined whether the target message is a duplicate message to avoid repeated processing or message disorder. Furthermore, the consumer end can ignore invalid messages and duplicate messages, and ensure that the same message is consumed only once while ensuring data consistency. Second, the message queue can determine which messages are in the next batch through the maximum subscript value of the messages that have been pushed successfully; by querying whether there is a target batch processing mark in the cache server, it can be determined whether the target batch messages have been successfully pushed. This avoids repeated pushing of target batch messages and triggers alarms in time to troubleshoot when push fails. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0080] Figure 1 A schematic diagram of the structure of a data processing system based on a message queue provided by an embodiment of the present invention;

[0081] Figure 2 A flowchart of a data processing method with a consumer end as the execution subject provided by an embodiment of the present invention;

[0082] Figure 3 A specific flow chart of a consumer end processing a single message provided in an embodiment of the present invention;

[0083] Figure 4 A specific flow chart of a consumer end processing batch messages provided by an embodiment of the present invention;

[0084] Figure 5 A flowchart of a data processing method using a message queue as an execution subject provided by an embodiment of the present invention;

[0085] Figure 6 A specific flow chart of a message queue pushing messages to a consumer end provided by an embodiment of the present invention;

[0086] Figure 7 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0088] The embodiments of the present invention provide a data processing method, system and server based on a message queue. Figure 1 The data processing system based on message queue provided by the present invention may include a message queue, a consumer end and a cache server. Among them, the message queue can be created based on RocketMQ or Kafka technology, or it can be created based on other technologies, and the present invention does not limit this. The consumer end may include more than two consumers, and only one of the consumers can consume the message at the same time, and the other consumers are in a standby state. If the consumer responsible for consuming the message fails, it can be switched to a standby consumer in time to consume the message. Each consumer can be understood as any network device that can process messages or a cluster device composed of multiple network devices. The cache server can be any network device that can provide data storage and query functions or a cluster device composed of multiple network devices. The present invention realizes asynchronous communication between the production end and the consumer end through a message queue. For example, the message queue can receive a message sent by the production end and send the message to the consumer end, and can also return a corresponding response to the production end and receive a response from the consumer end.

[0089] It should be noted that the production end can be any producer. When the production end generates a new message, it can add a time-sequenced identification code (which can be called a message sequence number) to the message body (Body). For example, in a business scenario where absolute timing is required for messages, when generating a message, the production end can request a single global sequence server (that is, a network device that can generate ordered sequence numbers) to obtain a globally unique message sequence number that is absolutely ordered in time, and add the message sequence number to the message body. In addition, in business scenarios where small errors in message timing are allowed, each producer can also request a message sequence number from a distributed sequence server nearby, or generate a message sequence number by itself. Of course, each producer can also obtain a message sequence number for each message through other means according to actual conditions, and the present invention is not limited to this.

[0090] In one embodiment, in order to ensure that the message is not lost, each producer can usually repeatedly send the same message to the message queue. The conditions for triggering the retransmission mechanism and the number of retransmissions can be preset according to the situation, and the present invention does not limit this.

[0091] The following will be combined with a specific implementation method to provide a data processing method based on a message queue in detail. Figure 2 and Figure 3 .

[0092] Step 201, parse the target key field and the target message sequence number of the target message, and obtain the maximum message sequence number corresponding to the target key field from the cache server.

[0093] In one embodiment, multiple messages related to each other may have the same key field (which may be recorded as key). The messages received by the consumer end from the message queue all carry a message sequence number, and the message sequence number of each message is an identification code with a time sequence, and different messages have different message sequence numbers. For ease of description, messages with the same key field may be referred to as similar messages, and any message received by the consumer end may be referred to as a target message. Accordingly, the key field carried by the target message may be referred to as a target key field, and the message sequence number carried by the target message may be referred to as a target message sequence number (which may be recorded as seq_recieve).

[0094] Every time the consumer successfully consumes the latest received message, it can record the mapping relationship item between the key field of the message and the message sequence number of the message in the cache server. In this way, the consumer can search for the mapping relationship item according to the target key field of the target message, and then obtain the maximum message sequence number (remembered as max_seq) corresponding to the target key field stored in the cache server. Based on the maximum message sequence number, it can be determined whether the consumer has consumed any message with the same target key field as the target message, and determine the timing relationship between the target message and the same type of messages that have been consumed.

[0095] For example, a producer (e.g., a user terminal) sends a first nickname request and a second nickname request to a designated message queue, respectively, to instruct the consumer to modify the user nickname to the first nickname and the second nickname in sequence. The first nickname request and the second nickname request both correspond to the same user terminal, so the key fields of the first nickname request and the second nickname request may be a user identification code (which may be referred to as userID), that is, the first nickname request and the second nickname request are associated with each other through the user identification code. The message sequence numbers of the first nickname request and the second nickname request may reflect the timing relationship between the first nickname request and the second nickname request.

[0096] Of course, in one embodiment, similar messages may also come from different producers (e.g., different user terminals). Taking the modification of the group chat name as an example, the first user terminal and the second user terminal respectively send a first group chat name request and a second group chat name request to the designated message queue. The first group chat name request and the second group chat name request both correspond to the same chat group, so the key fields of the first group chat name request and the second group chat name request may be an identification code for uniquely identifying the chat group.

[0097] Optionally, the consumer end may obtain messages one by one or in batches from the message queue. When the consumer end obtains messages in batches from the message queue, the processing before step 201 may include: obtaining target batch messages from the message queue, and querying the cache server whether there is a target batch processing mark; wherein the target batch message includes the target message; if there is, discarding the target batch message; if not, recording the target batch processing mark in the cache server and processing the target batch message.

[0098] In one embodiment, before consuming each batch of messages, the consumer can record the processing mark corresponding to the batch of messages in the cache server. For ease of description, any batch of messages obtained by the message queue can be called a target batch of messages, and correspondingly, the processing mark corresponding to the target batch of messages can be called the target batch processing mark.

[0099] like Figure 4As shown, after obtaining the target batch message, the consumer can query the cache server whether the target batch processing mark is recorded. If there is no record, it means that the target batch message is likely not obtained by any consumer. At this time, the target batch processing mark can be recorded in the cache server and each message in the target batch message can be processed. If there is a record, it means that the target batch message has been obtained by any consumer and may have been successfully consumed. At this time, the target batch message can be discarded. Based on the target batch processing mark, the consumer can discard the entire batch of messages after determining that the target batch message has been obtained by any consumer. In this way, there is no need to process the messages in the target batch message one by one, saving computing resources on the consumer side. Among them, taking the target batch message as the 1st to 100th message received by the message queue as an example, the target batch processing mark can be recorded as PROCESSING: [0-99].

[0100] It is worth mentioning that there are two ways for consumers to get messages from the message queue. Specifically, when the push method is adopted, the message queue actively pushes the message to the consumer; when the pull method is adopted, the consumer actively pulls the message from the message queue.

[0101] Optionally, the processing after the consumer end pulls the target batch of messages from the message queue may also include: recording the maximum index value of the target batch of messages in the cache server to pull the next batch of messages according to the maximum index value.

[0102] Optionally, the processing after the consumer end receives the target batch messages pushed by the message queue may also include: replying a successful reception response to the message queue, and the message queue recording the maximum subscript value of the target batch messages in the cache server, so that the message queue pushes the next batch of messages according to the maximum subscript value.

[0103] In one embodiment, each time a message queue receives a message (including new messages generated by the producer and resent messages), a message index (which can be recorded as offset) is set for the message. The corresponding message can be located by the message index. Taking the consumer actively pulling 100 messages from the message queue as an example, the target batch message can be the 1st to 100th messages received by the message queue. Among them, the message index increases from 0, the message index of the first message received by the message queue can be [0], and the message index of the 100th message can be

[99] . After the consumer pulls the target batch message with a message index range of [0-99] from the message queue, the maximum index value of the target batch message can be recorded in the cache server (which can be recorded as MAX_PULL_OFFSET=99). In this way, after obtaining the target batch message, the consumer can continue to pull the next batch of messages after the 100th message according to the maximum index value MAX_PULL_OFFSET=99 recorded in the cache server, that is, pull messages with a message index range of [100-109]. Of course, when there are less than 100 messages in the message queue, the message queue can provide all the messages currently queued in the message queue to the consumer when the consumer pulls the message.

[0104] Optionally, when the push method is used, the message queue actively pushes messages to the consumer. For details on how to push messages from the message queue, see Figure 5 and Figure 6 .

[0105] Step 501, query the maximum index value of the successfully pushed messages, and push the target batch of messages to the consumer end according to the maximum index value.

[0106] In one embodiment, before pushing any batch of messages, the message queue can query the cache server for the maximum subscript value of the messages that have been successfully pushed, and push the messages after the message corresponding to the maximum subscript value to the consumer end. Each batch of messages contains at least one message. It can be understood that if the message queue has not pushed any message to any consumer end, the maximum subscript value will not be stored in the cache server or the stored maximum subscript value is empty. The message queue can push messages to the consumer end starting from the first message when the maximum subscript value is not queried, or the message queue can directly push the first batch of messages to the consumer end without querying the maximum subscript value.

[0107] Step 502: If a successful reception response is received from the consumer end, or if it is determined that the target batch is being processed in the cache server, the target batch message is successfully pushed, and the maximum subscript value of the target batch message is recorded.

[0108] In one embodiment, after receiving the message pushed by the message queue, the consumer returns a corresponding successful reception response to the message queue. The message queue can determine whether the corresponding message has been successfully pushed by judging whether the successful reception response is received. When a successful reception response corresponding to the target batch message is received, the maximum subscript value of the target batch message can be recorded in the cache server, and then the next batch of messages can be pushed according to the maximum subscript value.

[0109] In one embodiment, due to network anomalies between the consumer and the message queue, the consumer successfully receives the message pushed by the message queue, but the message queue does not receive a successful reception response returned by the consumer. The message queue may push the target batch message to the consumer again. Therefore, in order to more accurately determine whether the target batch message is pushed successfully, the message queue can further determine whether there is a target batch processing mark in the cache server. If there is a target batch processing mark, it means that the consumer is already processing the target batch message, and the target batch message has undoubtedly been pushed successfully. At this time, the maximum subscript value of the target batch message can also be recorded in the cache server to avoid repeatedly pushing the same batch of messages to the consumer.

[0110] For example, the subscript range of the target batch message is [0-99], and the successful reception response corresponding to the target batch message may be a successful reception response carrying the subscript range [0-99]. When receiving a successful reception response with the subscript range [0-99], the message queue may record the maximum subscript value MAX_PUSH_OFFSET=99 of the target batch message in the cache server.

[0111] It is worth mentioning that when the message queue determines that the target batch message push is successful, it can also record the subscript range of the successfully pushed message to facilitate message analysis and statistics. For example, when a successful reception response of the subscript range [0-99] is received or the target batch processing mark bit PROCESSING: [0-99] exists in the cache server, PUSH_SUCCESS: [0-99] is recorded in the cache server. When a successful reception response of the subscript range [100-109] is continued to be received or the processing mark bit PROCESSING: [100-109] is queried, the PUSH_SUCCESS: [0-99] already recorded in the cache server can be updated to PUSH_SUCCESS: [0-109].

[0112] Step 503: If no successful reception response is received within the third preset time period, and there is no target batch processing mark in the cache server, it is determined that the target batch message push fails and an alarm is triggered.

[0113] In one embodiment, the message queue may start timing when sending the target batch message. If the successful reception response corresponding to the target batch message returned by the consumer end is not received within a preset time interval (which may be referred to as a third preset time length), the target batch message may fail to be pushed. Further, if there is no target batch processing mark in the cache server, it means that no consumer end is processing the target batch message. At this time, it can be determined that the target batch message has failed to be pushed and an alarm is triggered.

[0114] It is worth mentioning that before pushing the target batch message to the consumer end, the message queue can query the cache server whether there is a target batch push mark. If not, the message queue can record the target batch push mark in the cache server, and delete the target batch push mark after determining that the target batch message is pushed successfully. Pushing the target batch push mark reflects that the message queue is pushing or has pushed the target batch message. Recording the target batch push mark in the cache server facilitates message analysis and statistics. For example, the subscript range of the target batch message is [0-99]. Before pushing the target batch message, the target batch push mark PUSHING: [0-99] can be recorded in the cache server.

[0115] After step 201, in order to determine whether the target message to be consumed by the consumer is an invalid message or a duplicate message, before processing the target message, the consumer can determine the consumption progress of the same type of message by comparing the target message sequence number with the maximum message sequence number of the same type of message recorded in the cache server. For example: if the maximum message sequence number is greater than the target message sequence number, it means that the target message is an invalid message; if the maximum message sequence number is equal to the target message sequence number, it means that the target message is a duplicate message; if the maximum message sequence number is less than the target message sequence number, it means that the target message is a new message compared to the message that has been successfully consumed.

[0116] In one embodiment, the consumer responsible for consuming messages at the consumer end may fail after successfully consuming the target message and fail to delete the target message processing mark in time. The message queue may send the target message repeatedly to the consumer end. In order to avoid other consumers at the consumer end from repeatedly consuming the target message. After determining that the maximum message sequence number is less than the target message sequence number, it can be further determined whether there is a target message processing mark in the cache server. If there is no target message processing mark, it can be considered that the target message is a new message that has not been consumed by any consumer; if there is a target message processing mark, it can be said that a consumer is already processing or has successfully consumed the target message, and the target message is a duplicate message. At this time, the target message can be ignored.

[0117] Correspondingly, the processing of new messages that have not been consumed by any consumer can refer to step 202, and the processing of invalid messages and duplicate messages can refer to step 203.

[0118] Step 202: If the maximum message sequence number is less than the target message sequence number and the target message processing flag does not exist in the cache server, the consumer end records the target message processing flag in the cache server and consumes the target message.

[0119] In one embodiment, before consuming each message, the consumer end may record the processing mark corresponding to the message in the cache server. For ease of description, the processing mark corresponding to the target message may be referred to as the target message processing mark. The target message processing mark may include the target key field and the target message sequence number (which can be recorded as key+seq_receive=true). The target message processing mark may also include the unique identity of the consumer processing the target message (which can be recorded as key+seq_receive+consumerID=true), which makes it easier to view the consumer responsible for consuming the target message, and then to count and analyze the message consumption, and accurately locate the abnormal consumer.

[0120] Step 203: If the maximum message sequence number is not less than the target message sequence number, or the maximum message sequence number is less than the target message sequence number and there is a target message processing flag in the cache server, the target message is ignored.

[0121] In one embodiment, a certain user terminal sends a first nickname request at 10:10:10 (which can be recorded as 10:10:10), and sends a second nickname request at 10:10:11 (which can be recorded as 10:10:11). Due to factors such as network instability between the user terminal and the message queue or between the message queue and the consumer end, the order of the first nickname request and the second nickname request received by the consumer end is reversed. At this time, the consumer end will consume the second nickname request first, and then consume the first nickname request. In this way, the user nickname is finally set to the first nickname by the consumer end, which is inconsistent with the second nickname that the user terminal expects to set. Or, due to the retransmission mechanism of the user terminal, the consumer end receives the second nickname request twice, causing the consumer end to repeatedly consume the second nickname request, wasting computing resources. Therefore, the consumer end can use step 203 to ignore duplicate messages and invalid messages.

[0122] Generally speaking, ignoring the target message can be understood as discarding the target message. In some cases, for duplicate messages, ignoring the target message can be understood as temporarily not processing the target message and storing the target message in a preset waiting queue to wait for the consumer to obtain it from the waiting queue and determine again whether to consume the target message.

[0123] Still taking a user terminal sending out the first nickname request and the second nickname request successively as an example, because the consumer terminal uses multi-threaded parallel processing of messages or due to other circumstances, the cache server may record the first nickname request processing mark and the second nickname request processing mark at the same time, and the subsequent consumer terminal may first successfully consume the second nickname request, and then successfully consume the first nickname request. In the case where the consumer terminal already has the second nickname request processing mark in the cache server, or in the case where the second nickname request has been successfully consumed and the first nickname request is being consumed, the second nickname request obtained again can be temporarily stored in the waiting queue instead of being directly discarded. Because if the second nickname request obtained again is directly discarded, the user nickname will eventually be set to the first nickname by the consumer terminal, which is inconsistent with the second nickname that the user terminal expects to set. The second nickname request obtained again is temporarily stored in the waiting queue, so that after the consumer terminal successfully consumes the first nickname request, the second nickname request can be obtained and consumed again from the waiting queue, thereby ensuring data consistency.

[0124] Step 204: When the target message is consumed successfully, the consumer updates the maximum message sequence number stored in the cache server with the target message sequence number.

[0125] In one embodiment, after the consumer successfully consumes the target message, the maximum message sequence number stored in the cache server can be updated in a timely manner. For example, a mapping relationship item between the target key field of the target message and the target message sequence number can be added to the cache server, or the message sequence number in the mapping relationship item of the same type of message already recorded by the cache server can be replaced with the target message sequence number.

[0126] Optionally, each time the consumer successfully consumes a message, the processing mark corresponding to the single message stored in the cache server may be deleted. Accordingly, the processing of step 204 may also include: when the target message is consumed successfully, deleting the processing mark of the target message; if the target message processing mark is stored in the cache server for more than a first preset time, triggering an alarm.

[0127] In one embodiment, the cache server can start timing when the consumer end records the mark bit in the target message processing. If the consumer end does not delete the mark bit in the target message processing within a preset time interval (which can be called the first preset time length), it means that there is an abnormality in the consumption process of the target message. At this time, an alarm can be triggered to facilitate troubleshooting.

[0128] Optionally, if there is a problem such as an abnormality in the target message itself, the normal consumer end will not be able to successfully consume the target message. At this time, the consumer end can actively move the target message that failed to be consumed to the abnormal message queue. Accordingly, the data processing method provided by the present invention may also include: when the target message consumption fails, moving the target message to the abnormal message queue; if any message exists in the abnormal message queue, triggering an alarm.

[0129] In one embodiment, the data processing system may further include a monitoring server (not shown in the drawings), and the monitoring server is used to monitor the abnormal message queue in real time. Once any message is found to be stored in the abnormal message queue, an alarm may be issued through channels such as emails and text messages to notify technicians to troubleshoot. Of course, in order to avoid frequent alarms, other conditions for triggering alarms may be set on the monitoring server, and the present invention is not limited to this.

[0130] Optionally, in order to avoid repeated consumption of target batch messages by the consumer end, the data processing method provided by the present invention may also include: when all message processing in the target batch messages is completed, deleting the mark bit in the target batch processing; if the storage time of the mark bit in the target batch processing in the cache server exceeds a second preset time length, triggering an alarm.

[0131] In one embodiment, a problem may occur, for example, that after the consumer end records the target batch processing mark bit in the cache server, a failure occurs before the target batch message has been successfully consumed. In this case, the target batch processing mark bit will continue to be stored in the cache server. After switching the consumer end, since the target batch processing mark bit exists in the cache server, the new consumer end may discard the target batch message even if it obtains it. Therefore, if the target batch processing mark bit is stored in the cache server for more than a preset time interval (which may be referred to as a second preset duration), it can be determined that the target batch message consumption is abnormal and an alarm is triggered.

[0132] It is worth mentioning that the data processing system can also use the monitoring server to monitor the cache server to monitor anomalies in time, and then issue an alarm in time to eliminate the fault. For example, the first preset time and the second preset time can both be monitored by the monitoring server. Specifically, when the survival time of the mark bit in the cache server in the single message processing corresponding to any message exceeds the first preset time, when the survival time of the mark bit in the cache server in the batch message processing corresponding to any batch of messages exceeds the second preset time, the monitoring server can trigger an alarm.

[0133] It should be noted that the values ​​of the first preset time, the second preset time and the third preset time may be the same or different, and may be preset in the corresponding network device based on experience.

[0134] Compared with the prior art, the embodiments of the present invention can at least bring the following beneficial effects: First, the consumer end can determine whether the target message is an invalid message or a duplicate message by comparing the target message sequence number with the maximum message sequence number stored in the cache server; by querying whether there is a target message processing mark in the cache server, it can be further determined whether the target message is a duplicate message to avoid repeated processing or message disorder. Furthermore, the consumer end can ignore invalid messages and duplicate messages, and ensure that the same message is consumed only once while ensuring data consistency. Second, the message queue can determine which messages are in the next batch through the maximum subscript value of the messages that have been pushed successfully; by querying whether there is a target batch processing mark in the cache server, it can be determined whether the target batch messages have been successfully pushed. This avoids repeated pushing of target batch messages and triggers alarms in time to troubleshoot when push fails.

[0135] Based on the same technical concept, an embodiment of the present invention further provides a data processing system based on a message queue, the system comprising a message queue and a consumer end, wherein the consumer end is used to:

[0136] Parsing a target key field and a target message sequence number of a target message obtained from a message queue, and obtaining a maximum message sequence number corresponding to the target key field from a cache server; wherein the target message sequence number is an identification code with a time sequence;

[0137] If the maximum message sequence number is not less than the target message sequence number, or the maximum message sequence number is less than the target message sequence number and there is a target message processing flag in the cache server, then the target message is ignored;

[0138] If the maximum message sequence number is less than the target message sequence number and the target message being processed does not exist in the cache server, the target message being processed is recorded in the cache server and the target message is consumed;

[0139] When the target message is consumed successfully, the maximum message sequence number stored in the cache server is updated with the target message sequence number.

[0140] Optionally, before parsing the target key field and the target message sequence number of the target message obtained from the message queue, the consumer end is further used to:

[0141] Obtain a target batch of messages from a message queue, and query the cache server whether there is a target batch being processed mark; wherein the target batch of messages includes the target message;

[0142] If it exists, the target batch message is discarded; if it does not exist, the target batch processing mark is recorded in the cache server.

[0143] Optionally, the consumer end pulls the target batch of messages from the message queue or receives the target batch of messages pushed by the message queue; wherein,

[0144] After pulling the target batch of messages from the message queue, the consumer end is further used to:

[0145] Recording the maximum index value of the target batch of messages in the cache server, so as to pull the next batch of messages according to the maximum index value;

[0146] After receiving the target batch message pushed by the message queue, the consumer end is further used to:

[0147] A successful reception response is replied to the message queue, and the message queue records the maximum index value of the target batch of messages in the cache server, so that the message queue pushes the next batch of messages according to the maximum index value.

[0148] Optionally, the consumer end is further used for:

[0149] When the target message is consumed successfully, the target message processing mark is deleted;

[0150] If the target message processing mark is stored in the cache server for a time period exceeding a first preset time period, an alarm is triggered.

[0151] Optionally, the consumer end is further used for:

[0152] When the target message consumption fails, the target message is moved to the abnormal message queue;

[0153] If there is any message in the abnormal message queue, an alarm is triggered.

[0154] Optionally, the consumer end is further used for:

[0155] When all messages in the target batch of messages are processed, the target batch processing mark is deleted;

[0156] If the storage time of the mark bit in the target batch processing in the cache server exceeds the second preset time length, an alarm is triggered.

[0157] Optionally, the message queue is used for:

[0158] Querying the maximum subscript value recorded in the cache server, and pushing a target batch message to the consumer end according to the maximum subscript value recorded in the cache server;

[0159] If the successful reception response is received, or it is determined that the target batch is being processed in the cache server, it is determined that the target batch message is pushed successfully, and the maximum subscript value of the target batch message is recorded in the cache server;

[0160] If the successful reception response is not received within the third preset time period, and the target batch processing mark does not exist in the cache server, it is determined that the target batch message push fails and an alarm is triggered.

[0161] It should be noted that the data processing system and the data processing embodiment provided in the above embodiment belong to the same concept, and the specific implementation process and the technical effects produced are detailed in the method embodiment, which will not be repeated here.

[0162] Figure 7 7 is a structural diagram of a server provided by an embodiment of the present invention. The server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units 722 (for example, one or more processors) and memories 732, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 742 or data 744. Among them, the memories 732 and the storage media 730 may be temporary storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the text annotation device 700. Furthermore, the central processing unit 722 may be configured to communicate with the storage medium 730 and execute a series of instruction operations in the storage medium 730 on the server 700.

[0163] The server 700 may further include one or more power supplies 729, one or more wired or wireless network interfaces 750, one or more input and output interfaces 758, one or more keyboards 756, and / or one or more operating systems 741, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0164] The server 700 may include a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for pushing the streaming media data.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the software product that implements the data processing method based on the message queue can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including storing a number of instructions to enable a network device to execute the methods described in each embodiment or some parts of the embodiment.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method based on a message queue, applied to a consumer end, characterized in that: The method comprises: Parsing a target key field and a target message sequence number of a target message obtained from a message queue, and obtaining a maximum message sequence number corresponding to the target key field from a cache server; wherein the target message sequence number is an identification code with a time sequence; Determine whether to consume the target message according to the size relationship between the maximum message sequence number and the target message sequence number and whether the target message processing flag is stored in the cache server; When the target message is consumed successfully, the maximum message sequence number stored in the cache server is updated with the target message sequence number; According to the size relationship between the maximum message sequence number and the target message sequence number, and whether the target message processing flag is stored in the cache server, determining whether to consume the target message includes: If the maximum message sequence number is greater than the target message sequence number, the target message is ignored; the maximum message sequence number is the message sequence number of the latest message in the message corresponding to the target key field.

2. The method according to claim 1, characterized in that According to the size relationship between the maximum message sequence number and the target message sequence number, and whether the target message processing flag is stored in the cache server, determining whether to consume the target message includes: If the maximum message sequence number is less than the target message sequence number and the target message being processed flag does not exist in the cache server, the target message being processed flag is recorded in the cache server and the target message is consumed; If the maximum message sequence number is equal to the target message sequence number, or the maximum message sequence number is less than the target message sequence number and there is a target message processing flag in the cache server, the target message is ignored.

3. The method according to claim 1, characterized in that Before parsing the target key field and the target message sequence number of the target message obtained from the message queue, the method further includes: Obtain a target batch of messages from a message queue, and query the cache server whether there is a target batch being processed mark; wherein the target batch of messages includes the target message; If it exists, the target batch message is discarded; if it does not exist, the target batch processing mark is recorded in the cache server.

4. The method according to claim 3, characterized in that The acquiring the target batch of messages from the message queue includes pulling the target batch of messages from the message queue or receiving the target batch of messages pushed by the message queue; After pulling the target batch of messages from the message queue, the method further includes: Recording the maximum index value of the target batch of messages in the cache server, so as to pull the next batch of messages according to the maximum index value; After receiving the target batch of messages pushed by the message queue, the method further includes: A successful reception response is replied to the message queue, and the message queue records the maximum index value of the target batch of messages in the cache server, so that the message queue pushes the next batch of messages according to the maximum index value.

5. The method according to claim 1, characterized in that The method further comprises: When the target message is consumed successfully, the target message processing mark is deleted; If the target message processing mark is stored in the cache server for a time period exceeding a first preset time period, an alarm is triggered.

6. The method according to claim 1, characterized in that The method further comprises: When the target message consumption fails, the target message is moved to the abnormal message queue; If there is any message in the abnormal message queue, an alarm is triggered.

7. The method according to claim 1 or 2, characterized in that: Ignoring the target message includes: The target message is stored in a waiting queue, so that the consumer end obtains the target message from the waiting queue and determines again whether to consume the target message.

8. The method according to claim 3, characterized in that The method further comprises: When all messages in the target batch of messages are processed, the target batch processing mark is deleted; If the storage time of the mark bit in the target batch processing in the cache server exceeds the second preset time length, an alarm is triggered.

9. A data processing method based on a message queue, applied to a message queue, characterized in that: The method comprises: Query the maximum subscript value of the message that has been pushed successfully, and push the target batch message containing the target message to the consumer end according to the maximum subscript value, so that the consumer end can parse the target key field and the target message sequence number of the target message, the target message sequence number is an identification code with a time sequence, and obtain the maximum message sequence number corresponding to the target key field from the cache server, the maximum message sequence number is the message sequence number of the message with the latest time sequence in the message corresponding to the target key field; the consumer end also determines whether to consume the target message according to the size relationship between the maximum message sequence number and the target message sequence number, and whether the target message processing mark is stored in the cache server, including: if the maximum message sequence number is greater than the target message sequence number, the consumer end ignores the target message; If a successful reception response from the consumer end is received, or if it is determined that the target batch is being processed in the cache server, it is determined that the target batch message is pushed successfully, and the maximum subscript value of the target batch message is recorded; If the successful reception response is not received within the third preset time period, and the target batch processing mark does not exist in the cache server, it is determined that the target batch message push fails and an alarm is triggered; The target batch processing mark is recorded in the cache server by the consumer end when the consumer end receives the target batch message.

10. The method according to claim 9, characterized in that The method further comprises: Each message in the target batch of messages carries a message sequence number, and the message sequence number of each message is an identification code with a time sequence; The message sequence number is used to enable the consumer to determine whether to consume the message corresponding to the message sequence number.

11. A data processing system based on a message queue, the system comprising a message queue and a consumer end, characterized in that: The consumer end is used to implement the method according to any one of claims 1 to 8, and / or the message queue is used to implement the method according to any one of claims 9 to 10.

12. A server, characterized in that: The server includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8, or to implement the method according to any one of claims 9 to 10.

13. A cache server, characterized in that: The cache server, the message queue and the consumer end jointly implement the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 8, or to implement the method as described in any one of claims 9 to 10.

Images