Message processing method, device, electronic device and storage medium

Abstract

The present application provides a message processing method, device, electronic device and storage medium, which relates to the field of computer technology and is used to ensure the sequentiality and concurrency of message processing. The method includes: continuously obtaining each message generated by a message source from at least one message queue, and saving each obtained message to at least one target queue each time; wherein, messages with the same identifier in each message are saved to the same target queue, and each message with the same identifier in a target queue is saved according to the arrangement order of each message with the same identifier in the corresponding message queue; for each target queue in at least one target queue, a processing thread is used to process each message in the corresponding target queue in the order in which the messages are saved, and obtain the message processing record corresponding to each message; based on each obtained message processing record, the message processing result is obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a message processing method, device, electronic device and storage medium. Background Art

[0002] With the development of Internet technology, the number of messages generated by message sources has increased exponentially, and the message processing end needs to process a large number of messages generated by the message sources. At present, when the message processing end obtains a large number of messages generated by the message source, it usually stores these messages in multiple message queues respectively, and then pulls messages from multiple message queues for processing in batches when processing these messages.

[0003] For a batch of pulled messages, multiple threads are used to process them simultaneously. It is necessary to wait until this batch of messages is processed before processing the next batch of messages. However, some messages of the same type generated by the message source may be ordered, such as messages from the same account or the same device. When multiple threads are used to process them simultaneously, it cannot be guaranteed that the ordered messages will be processed in order. In addition, if a thread is stuck, other threads will have to wait, resulting in poor concurrency. Summary of the invention

[0004] The embodiments of the present application provide a message processing method, device, electronic device and storage medium for ensuring the sequentiality and concurrency of message processing.

[0005] In a first aspect, an embodiment of the present application provides a message processing method, including:

[0006] Continuously obtaining each message generated by a message source from at least one message queue, and saving each obtained message into at least one target queue each time the messages are obtained; wherein messages with the same identifier among the messages are saved into the same target queue, and the messages with the same identifier in a target queue are saved in the order in which the messages with the same identifier are arranged in the corresponding message queue;

[0007] For each target queue in the at least one target queue, using a processing thread to process each message in the corresponding target queue in the order in which the messages are saved, and obtaining a message processing record corresponding to each message;

[0008] Based on each obtained message processing record, a message processing result is obtained.

[0009] In a second aspect, an embodiment of the present application provides a message processing device, including:

[0010] A saving module, used for continuously acquiring each message generated by a message source from at least one message queue, and saving each acquired message to at least one target queue each time; wherein, messages with the same identifier among the messages are saved to the same target queue, and the messages with the same identifier in a target queue are saved according to the arrangement order of the messages with the same identifier in the corresponding message queue;

[0011] A processing module, configured to process each message in the corresponding target queue using a processing thread according to the order in which the messages are saved, for each target queue in the at least one target queue, and obtain a message processing record corresponding to each message;

[0012] The acquisition module is used to obtain the message processing result based on each message processing record obtained.

[0013] In a possible embodiment, each message obtained from at least one message queue carries a message queue identifier, and the device further includes:

[0014] A determination module, for determining, for each message processing record corresponding to each currently processed message, the message processing record of each message from the corresponding message queue according to the message queue identifier carried by each processed message;

[0015] The sorting module is used to arrange the message processing records of each message in the corresponding message queue according to the order of each message in the message queue to obtain the arranged message processing records corresponding to the corresponding message queue.

[0016] In a possible embodiment, the message processing record includes an offset of the corresponding message in the corresponding message queue, and the offsets of the messages obtained from the same message queue are continuously increased, and the device further includes:

[0017] A determination module, configured to determine an offset sequence for a corresponding message queue according to an offset in an arranged message processing record corresponding to the message queue;

[0018] The recording module is used to determine the maximum offset in the first continuous offset subsequence in the offset sequence, and record the message corresponding to the maximum offset that has been processed in the corresponding message queue.

[0019] In a possible embodiment, the device further includes:

[0020] A deletion module is used to delete the message processing records corresponding to the offsets other than the maximum offset in the first continuous offset subsequence.

[0021] In a possible embodiment, the device further includes:

[0022] an alarm module, configured to generate an alarm if the maximum offset in the offset sequence changes within a set time and the maximum offset in the first continuous offset subsequence does not change; or,

[0023] If the maximum offset in the offset sequence increases by a set value and the maximum offset in the first continuous offset subsequence does not change, an alarm is issued; or,

[0024] If the maximum offset in the first continuous offset quantum sequence does not change, and the difference between the maximum offset in the offset sequence and the maximum offset in the first continuous offset quantum sequence reaches a set threshold, an alarm is issued.

[0025] In a possible embodiment, the storage module is further used for:

[0026] Performing hash operations on the identifiers of the respective messages to obtain corresponding hash values;

[0027] Each of the messages is saved in a target queue associated with a corresponding hash value.

[0028] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory, wherein the memory stores a program code, and when the program code is executed by the processor, the processor executes the steps of the method described in the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the steps of the method described in the first aspect.

[0030] Since the embodiment of the present application adopts the above technical solution, it has at least the following technical effects:

[0031] Continuously obtain each message generated by the message source from multiple message queues, and each time a message is obtained, save each obtained message into multiple target queues respectively, and the messages with the same identifier in each message are saved into the same target queue according to the arrangement order in the corresponding message queue, wherein the messages with the same identifier can be messages of the same type, so that when the messages with the same identifier are ordered messages, the ordered messages can maintain their order. For each target queue, a processing thread is used to process each message in the target queue according to the order in which the messages are saved, so that the ordered messages can be processed in order to ensure their processing order.

[0032] In addition, since multiple target queues are processed by one processing thread respectively, multiple processing threads process messages concurrently, and one of the processing threads does not need to wait for other processing threads to complete processing, but can continuously process messages. Therefore, compared with the method of batch-pulling messages in the related art, waiting for the processing of this batch of messages before processing the next batch of messages, the concurrency of the embodiment of the present application is better. Therefore, the embodiment of the present application can ensure the processing sequence of ordered messages, and has good concurrency.

[0033] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A flowchart of message processing in the related technology provided by the embodiment of the present application;

[0036] Figure 2 It is a structural diagram of a blockchain system provided by an embodiment of the present application;

[0037] Figure 3 is a schematic diagram of a block structure provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of an application scenario of a message processing method provided in an embodiment of the present application;

[0039] Figure 5 A flowchart of a message processing method provided in an embodiment of the present application;

[0040] Figure 6 A flowchart of a message processing method of a Kafka-based message processing system provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of a message processing process of a Kafka-based message processing system provided in an embodiment of the present application;

[0042] Figure 8A schematic diagram of the processing process of the offset processing module in the Kafka-based message processing system provided in an embodiment of the present application;

[0043] Figure 9a A schematic diagram of a message processing process after a shutdown and restart in the related technology provided in an embodiment of the present application;

[0044] Figure 9b A schematic diagram of another message processing process after a shutdown and restart provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of the structure of a message processing device provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0048] In order to help those skilled in the art better understand the technical solution of the present application, the terms involved in the present application are introduced below.

[0049] 1. Cloud technology: refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data calculation, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing. The servers involved in the embodiments of this application can all be implemented through cloud technology.

[0050] 2. Message source: refers to the device that generates messages, such as terminal devices, which can generate business-related messages when processing business.

[0051] 3. Message queue: refers to a container that stores messages during the transmission process of the messages. The message queue in the embodiment of the present application can be deployed in the server to cache messages.

[0052] 4. Identification: refers to the identification of the message, which is used to distinguish different types of messages. For example, the identification of a message can be the device identification of the terminal device that generates the message, or the identification of the account that logs into the terminal device, or the service identification corresponding to the message. It can be set as needed. Therefore, different types of messages can be messages from different accounts, messages from different devices, or messages from different services, etc., and this application does not limit this.

[0053] 5. Thread: It is the smallest unit that the operating system can schedule operations. It is contained in the process and is the actual operation unit in the process. A thread refers to a single sequential control flow in the process. Multiple threads can run concurrently in a process, and each thread executes different tasks in parallel.

[0054] 6. Downtime: This means that the computer can no longer provide services due to some reasons. It is a situation in which the computer's computing power disappears.

[0055] 7. Kafka: It is an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data on the consumer side of the website. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time messages through clusters.

[0056] 8. Offset: The position of the message that has been read in the Kafka message log. In the embodiment of the present application, it can be understood as the position of each message obtained from at least one message queue.

[0057] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0058] The terms "first" and "second" in this document are used for descriptive purposes only and should not be understood as explicitly or implicitly indicating relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0059] The following is an introduction to the relevant technologies:

[0060] The following takes the process of a server processing a large number of messages generated by a terminal device as an example to introduce the message processing process in the related art:

[0061] For a large number of messages generated by terminal devices, the server first caches these messages and then uses Figure 1 The processing process shown in the figure is to pull messages from the cache in batches through the data pulling thread. For the pulled batch of messages, multiple threads such as thread 1, thread 2...thread n are used to process them. After all threads have processed this batch of messages, the next batch of messages are pulled for the same processing. However, some messages of the same type generated by the terminal device may be ordered. When multiple threads are used to process them at the same time, it cannot be guaranteed that they are processed in the order of ordered messages. Moreover, if a thread is stuck, other threads will be forced to wait, resulting in poor concurrency.

[0062] In view of this, an embodiment of the present application provides a message processing method, and the design concept of the method is introduced below.

[0063] In an embodiment of the present application, after the message processing end obtains a large number of messages generated by the message source, the message processing end can cache these messages into multiple message queues. Specifically, messages with the same identifier are stored in the same message queue in the order of acquisition, wherein the messages with the same identifier can be messages of the same type. When processing messages, messages can be obtained in batches from multiple message queues. Each time a message is obtained, the obtained messages are split twice, that is, they are saved in multiple target queues respectively, and the messages with the same identifier in each message are saved in the same target queue according to their arrangement order in the corresponding message queue. In this way, when the messages with the same identifier are ordered messages, the ordered messages can maintain their sequence.

[0064] For each target queue, a processing thread is used to process each message in the target queue in the order in which the messages are saved, so that the ordered messages can be processed in order to ensure the processing sequence. In addition, since multiple target queues are processed by a processing thread respectively, multiple processing threads process messages concurrently, and one of the processing threads does not need to wait for other processing threads, but can continuously process messages, and the concurrency is good. Therefore, the embodiment of the present application can ensure the processing sequence of ordered messages, and the concurrency is good.

[0065] In some embodiments, the above-mentioned message queues can be understood as blocks in the blockchain system. Figure 2 , Figure 2It is an optional structural diagram of the blockchain system 400 provided in the embodiment of the present application, which is formed by multiple nodes 401 (any form of computing devices in the access network, such as servers and user terminals) and clients 402. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In a distributed system, any machine such as a server or a terminal can join and become a node. The node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0066] See also Figure 2 The functions of each node in the blockchain system shown include:

[0067] 1) Routing: a basic function of a node, used to support communication between nodes.

[0068] In addition to the routing function, the node can also have the following functions:

[0069] 2) Applications are deployed in the blockchain to implement specific businesses based on actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to a temporary block when they successfully verify the source and integrity of the record data.

[0070] For example, the services implemented by the application include:

[0071] 2.1) Wallet, used to provide the function of conducting electronic currency transactions, including initiating transactions (i.e., sending the transaction record of the current transaction to other nodes in the blockchain system. After the other nodes successfully verify the transaction, as a response to acknowledge the validity of the transaction, the transaction record data is stored in the temporary block of the blockchain; of course, the wallet also supports querying the remaining electronic currency in the electronic currency address;

[0072] 2.2) Shared ledger, which is used to provide functions such as storage, query and modification of account data. The record data of the operation on the account data is sent to other nodes in the blockchain system. After other nodes verify the validity, as a response to acknowledging the validity of the account data, the record data is stored in a temporary block, and a confirmation can also be sent to the node that initiated the operation.

[0073] 2.3) Smart contracts are computerized protocols that can execute the terms of a contract. They are implemented by deploying code on a shared ledger that is executed when certain conditions are met. The code is used to complete automated transactions based on actual business needs, such as querying the logistics status of the goods purchased by the buyer and transferring the buyer's electronic currency to the merchant's address after the buyer signs for the goods. Of course, smart contracts are not limited to executing contracts for transactions, but can also execute contracts for processing received information.

[0074] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0075] See also Figure 3 , Figure 3 This is an optional schematic diagram of the block structure provided by the embodiment of the present application. Each block includes the hash value of the transaction record stored in this block (the hash value of this block) and the hash value of the previous block. Each block is connected by the hash value to form a blockchain. In addition, the block can also include information such as the timestamp when the block was generated. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains relevant information for verifying the validity of its information (anti-counterfeiting) and generating the next block.

[0076] Based on the above design ideas, the application scenarios of the message processing method of the embodiment of the present application are introduced below.

[0077] Combine the following Figure 1 A scenario diagram of a message processing method is shown, which provides an example introduction to the devices that may be involved in the application scenario of an embodiment of the present application.

[0078] Reference Figure 4 , the application scenario includes a terminal device 100, a cache server 200 and a processing server 300. The terminal device 100 and the cache server 200 can communicate through a communication network, and the cache server 200 and the processing server 300 can also communicate through a communication network. Optionally, the communication network can be a wired network or a wireless network. The terminal device 100 and the cache server 200, and the cache server 200 and the processing server 300 can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0079] The terminal device 200 may be, for example, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal device, etc., but is not limited thereto. The terminal device 110 may execute various services through a client, and the client may be a web version of a client, an application preinstalled in the terminal device 200, or a sub-application embedded in a third-party application, etc. When the terminal device 200 executes any service through the client, a service-related message may be generated.

[0080] The cache server 200 and the processing server 300 can both be a single server or a server cluster consisting of several servers. They can be virtual servers or physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0081] In a specific implementation, the cache server 200 can obtain the messages generated by the terminal device 100, and store the obtained messages in multiple message queues. Specifically, messages with the same identifier can be stored in the same message queue in the order of acquisition. The above message can be sent from the terminal device 100 to the cache server 200, or it can be obtained from the terminal device 100 by the cache server 200, and this application does not limit this. Furthermore, the processing server 300 can serve as a message processing end, and obtain messages from multiple message queues in the cache server 200 in batches for processing, wherein the message processing process involved will be introduced below.

[0082] It should be noted that Figure 4 This is an example of an application scenario of the message processing method, and the actual application scenario to which the method in the embodiment of the present application can be applied is not limited to this. For example, the above-mentioned multiple message queues can also be deployed in the processing server 300, that is, the message processing end can only include the processing server 300, and the embodiment of the present application does not limit this.

[0083] In a possible implementation scenario, the cache server 200 and the processing server 300 can form a message processing system based on Kafka. Among them, Kafka is a high-throughput distributed publish-subscribe message system, and the message processing system based on Kafka can include a message production end, a Kafka cluster composed of multiple servers, and a consumer end. The message production end and the Kafka cluster can be understood as the above-mentioned cache server 200, and the consumer end can be understood as the above-mentioned processing server 300. The message production end can obtain the message generated by the message source and push the message to one or more servers in the Kafka cluster. Specifically, messages with the same identifier can be pushed to the same server, each of which can deploy a queue shard (i.e., the above-mentioned message queue), and each server caches the obtained messages in the order of acquisition. The consumer end can pull the subscribed messages from one or more servers in the Kafka cluster, and then consume the pulled messages. It can be understood as continuously obtaining each message generated by the message source from at least one message queue and processing the obtained messages.

[0084] In practical applications, the message processing method of the embodiment of the present application can be applied to scenarios such as voting, payment, and liking that have high requirements on sequentiality and concurrency.

[0085] For example, in a live broadcast scenario, a large number of viewers can vote for the anchor through terminal devices (i.e., audience terminals), which can generate a large amount of voting data (i.e., messages). Each audience terminal can send the voting data to the cache server. After voting, the same audience may cancel the vote and then vote again. At this time, the multiple voting data of the same audience are in order. In addition, in a live broadcast scenario, a large number of viewers can also like the anchor through their respective audience terminals, which is similar to the above voting process and will not be repeated here. Therefore, this scenario has relatively high requirements for the order and concurrency of messages.

[0086] For example, in a payment scenario, users can conduct transactions through terminal devices, and billing data can be generated when making payments. A large number of users conduct transactions through terminal devices (i.e., user terminals), which can generate a large amount of billing data (i.e., messages). Each user's user terminal can send the billing data to the cache server. After a transaction, the same user may cancel the transaction and then conduct the transaction again. At this time, multiple billing data of the same user are in order. Therefore, this scenario has relatively high requirements for the order and concurrency of messages.

[0087] It should be noted that the above implementation scenarios are merely exemplary, and the implementation scenarios to which the message processing method in the embodiments of the present application can be applied are not limited to these.

[0088] Based on the above application scenarios, the message processing method of the embodiment of the present application is introduced below.

[0089] Figure 5 A flowchart of a message processing method provided in an embodiment of the present application is shown. The message processing method can be executed by a processing device, for example, Figure 4 The processing server 300 in Figure 5 As shown, the method may include the following steps:

[0090] Step S501, continuously obtain each message generated by the message source from at least one message queue, and save each message obtained into at least one target queue each time; wherein, messages with the same identifier in each message are saved into the same target queue, and messages with the same identifier in a target queue are saved in the order in which the messages with the same identifier are arranged in the corresponding message queue.

[0091] At least one message queue may be located in the cache server, for example Figure 2 The message source may be a terminal device that generates a message, and the terminal device may send the generated message to the cache server. In addition, the cache server may also actively obtain the message from the terminal device. It should be noted that at least one message queue may also be located in the processing server, which is not limited in the embodiments of the present application.

[0092] Specifically, the same identifier can be set for messages of the same type generated by the message source. Messages of the same type can be messages of the same account, messages of the same terminal device, messages of the same service, etc. Therefore, the above identifier can be an account identifier, a device identifier, or a service identifier, etc., which can be set according to the actual application scenario. Messages with the same identifier can be ordered messages in some scenarios.

[0093] In one example, when a user conducts a transaction through a client of a terminal device, after submitting a bill, a message related to the bill may be generated. Different users may generate different messages, and different messages may have different identifiers. For example, the identifier may be an account identifier of a logged-in client, or a device identifier or a bill identifier, etc. When a user submits a bill, cancels the bill, and then submits the bill again, three messages with the same identifier may be generated, and the three messages are in order.

[0094] In another example, in a live broadcast scenario, the audience can vote for the anchor through the audience client (hereinafter referred to as the audience terminal). At this time, the audience terminal, as a message source, can generate messages when the audience votes. Different audience terminals can generate different messages. Different messages can have different identifiers, which can be, for example, the account identifier of the logged-in audience terminal, or the device identifier of the audience terminal, etc. When a viewer votes and then cancels the vote, two messages with the same identifier can be generated, and the two messages are in order.

[0095] In specific implementation, after the cache server obtains each message generated by the message source, when allocating each message to different message queues, messages with the same identifier can be allocated to the same message queue, and each message queue can have a message queue identifier, which is used to distinguish different message queues, for example, message queue 1, message queue 2, message queue 3. When the messages with the same identifier are ordered messages, the order of the ordered messages is maintained in the message queue so that the ordered messages can be processed in their order later. In addition, the cache server can also set a unique ID for each message, that is, multiple messages may have the same identifier, for example, messages of the same type can have the same identifier, but each message corresponds to an ID, and each message can be distinguished by this ID.

[0096] Furthermore, the processing server may obtain messages from at least one message queue in batches. Specifically, the processing server may obtain messages from at least one message queue periodically. For example, after obtaining the first batch of messages, the second batch of messages may be obtained after a preset time t. After obtaining the second batch of messages, the third batch of messages may be obtained after a preset time t, and so on. Wherein t may be set as required, and the embodiments of the present application are not limited to this. Specifically, the number of messages obtained in each batch may be set as required, and the number of messages obtained in each batch may be the same or different.

[0097] In specific implementation, the same batch of messages can come from the same message queue or from different message queues. Exemplarily, multiple message queues include message queue 1, message queue 2, and message queue 3. When obtaining a batch of messages, a set number of messages can be obtained from any one of message queues 1, message queue 2, and message queue 3; a certain number of messages can also be obtained from message queue 1, message queue 2, and message queue 3, respectively, to obtain a set number of messages in total; of course, a certain number of messages can also be obtained from any two message queues of message queue 1, message queue 2, and message queue 3, respectively, to obtain a set number of messages in total. In addition, each message in the obtained batch of messages carries a message queue identifier, and the message queue to which each obtained message belongs can be determined based on the message queue identifier.

[0098] After the processing server obtains a batch of messages, it can distribute the messages to different target queues in the order of acquisition. For messages with the same identifier, they are saved in the same target queue. A target queue can include messages with multiple identifiers, and when a target queue saves messages with the same identifier, it saves them in the order in the corresponding message queue.

[0099] For example, in a certain batch of messages, including message 1, message 2, message 3 with identification 1, message 4, message 5, message 6 with identification 2, when saving the batch of messages, message 1, message 2, message 3 can be saved to the same target queue, and message 4, message 5, message 6 can be saved to the same target queue. The message with identification 1 and the message with identification 2 can be saved to the same target queue or to different target queues. Moreover, the order in which message 1, message 2, message 3 are saved is consistent with the order in which the three messages are in the corresponding message queues, and the order in which message 4, message 5, message 6 are saved is consistent with the order in which the three messages are in the corresponding message queues.

[0100] In some embodiments, in order to save the messages with the same identifier in the obtained messages into the same target queue, the messages obtained in step S501 are saved into at least one target queue respectively, which can be specifically implemented by the following steps:

[0101] Step a: Perform hash operations on the identifiers of each message to obtain corresponding hash values;

[0102] Step b: Save each message to the target queue associated with the corresponding hash value.

[0103] Specifically, different identifiers can be represented by different keys. For example, for a message containing key1, key1 can be hashed to obtain a corresponding hash value. Each target queue can be pre-set to be associated with a corresponding hash value. For example, the hash value associated with target queue 1 is H1, the hash value associated with target queue 2 is H2, and the hash value associated with target queue 3 is H3. When the hash value corresponding to the above key1 is H1, all messages containing key1 are saved in target queue 1. Thus, messages with the same identifier can be saved in the same target queue.

[0104] Step S502: for each target queue in at least one target queue, a processing thread is used to process each message in the corresponding target queue according to the order in which the messages are saved, and a message processing record corresponding to each message is obtained.

[0105] Exemplarily, the multiple target queues include target queue 1, target queue 2, and target queue 3, and the three target queues respectively use processing thread 1, processing thread 2, and processing thread 3 to process messages. Specifically, processing thread 1 processes each message in target queue 1 in the order in which the messages are saved, processing thread 2 processes each message in target queue 2 in the order in which the messages are saved, and processing thread 3 processes each message in target queue 3 in the order in which the messages are saved, and then the message processing records of the three processing threads can be obtained.

[0106] Taking processing thread 1 as an example, target queue 1 stores messages 1, 2, 3, 4, 5, and 6 in order. The processing thread processes messages 1, 2, 3, 4, 5, and 6 in order to obtain message processing records corresponding to the six messages.

[0107] Step S503: Obtain message processing results based on each message processing record obtained.

[0108] As can be seen from the above, each message has an ID, and the message processing record of each message can carry the message ID. In this way, it is possible to determine which messages have been processed based on the message IDs carried in the message processing record.

[0109] Exemplarily, in the above live broadcast scenario, for each message generated by the audience voting for the anchor through the audience terminal, after being processed through the above steps S501 and S502, the obtained message processing records can be the voting records corresponding to the processed messages, and then the voting results are counted according to the obtained voting records, that is, the message processing results are obtained.

[0110] In the embodiment of the present application, messages are obtained in batches from multiple message queues, and each time a message is obtained, each obtained message is saved in multiple target queues respectively, and messages with the same identifier in each message are saved in the same target queue according to the arrangement order in the corresponding message queue, so that when the messages with the same identifier are ordered messages, the ordered messages can maintain their order. For each target queue, a processing thread is used to process each message in the target queue according to the order in which the messages are saved, so that the ordered messages can be processed in order to ensure their processing order.

[0111] In addition, since multiple target queues are processed by one processing thread respectively, and multiple processing threads process messages concurrently, one of the processing threads does not need to wait for other processing threads, but can continuously process messages. Compared with the method of batch-pulling messages in the related art, waiting for the processing of this batch of messages before processing the next batch of messages, the concurrency of the embodiment of the present application is better. Therefore, the embodiment of the present application can ensure the processing sequence of ordered messages, and has good concurrency.

[0112] In some embodiments, since multiple processing threads process the acquired messages in parallel, the message processing records of the processed messages may be message processing records of messages from different message queues. After the above step S502, the following steps may be further performed:

[0113] Step 1) For the message processing records corresponding to each currently processed message, the message processing records of each message from the corresponding message queue are determined according to the message queue identifier carried by each processed message.

[0114] Step 2) Arrange the message processing records of each message in the corresponding message queue according to the order of each message in the message queue to obtain the arranged message processing records corresponding to the corresponding message queue.

[0115] Furthermore, the message processing record may include an offset of the corresponding message, which may indicate the position of a message in the corresponding message queue, for example, the offset of the first message in message queue 1 may be 1, the offset of the second message may be 2, and so on. If the acquired batch of messages comes from different message queues, the offset of each message is the offset in the corresponding message queue.

[0116] In the related art, when a processing server uses multiple threads to process messages, it will record the offset of the message currently processed by each message queue for each message queue, and usually record the maximum offset that has been processed, so that after the processing server crashes and restarts due to some reasons, it can continue to obtain messages after the maximum offset recorded in the message queue. However, this may result in the loss of messages. For example, the offsets of the processed messages in the corresponding message queue are: 1, 2, 3, 5, 6. At this time, if the record has processed the sixth message, it is considered that the messages before the sixth message have been processed. After the processing server crashes and restarts, it will continue to obtain the messages after the sixth message in the corresponding message queue. Since the fourth message has not been processed, the fourth message is lost. In order to prevent the loss of messages, the message processing method of the embodiment of the present application may also include the following steps:

[0117] Step 1: for a corresponding message queue, determine an offset sequence according to an offset in an arranged message processing record corresponding to the message queue;

[0118] Step 2: Determine the maximum offset in the first consecutive offset subsequence in the offset sequence, and record the message corresponding to the maximum offset that has been processed in the corresponding message queue.

[0119] For example, the arranged message processing records corresponding to message queue 1 include the message processing record of the first message, the message processing record of the second message, the message processing record of the third message, the message processing record of the fifth message, and the message processing record of the sixth message, and the corresponding offsets are 1, 2, 3, 5, and 6, respectively, that is, the obtained offset sequence is 1, 2, 3, 5, and 6.

[0120] For example, in the above offset sequence 1, 2, 3, 5, 6, the first continuous offset subsequence is 1, 2, 3, and the maximum offset is 3. That is to say, if the processing server crashes and restarts, the record shows that the third message has been processed. At this time, it is considered that the messages after the third message have not been processed, and the messages after the third message in the message queue 1 will continue to be obtained, and the messages after the third message will continue to be processed through the corresponding processing thread to prevent the loss of message 4.

[0121] The message processing method of the embodiment of the present application can not only ensure the sequentiality and concurrency of message processing, but also prevent message loss due to the shutdown and restart of the processing server.

[0122] Furthermore, the arranged message processing records corresponding to the corresponding message queues obtained in the above embodiment can be put into the cache. After recording the messages corresponding to the maximum offset in the corresponding message queue, in order to release the cache space, the message processing method of the embodiment of the present application can also include the following steps:

[0123] Delete the message processing records corresponding to the offsets except the maximum offset in the first continuous offset subsequence.

[0124] For example, the first continuous offset subsequence is 1, 2, 3, where the offsets except the maximum offset 3 are 1, 2. Therefore, the message processing record of the first message and the message processing record of the second message in the cache are deleted. This is because the first message and the second message have been processed, and the current processing has been recorded to the third message, so there is no need to record the message processing record of the first message and the message processing record of the second message. In this way, the cache space can be released.

[0125] Moreover, when recording the third message currently being processed, if the processing server crashes and restarts, at this time, the processing server will obtain the messages after the third message in the corresponding message queue. If the fifth and sixth messages are processed, since the message processing records of the fifth and sixth messages have been obtained before, the fifth and sixth messages will not be processed again, thereby avoiding duplicate processing of messages.

[0126] Further, after recording the message corresponding to the maximum offset that has been processed in the corresponding message queue, the message processing method of the embodiment of the present application may further include the following steps:

[0127] a) If the maximum offset in the offset sequence changes within the set time, and the maximum offset in the first consecutive offset subsequence does not change, an alarm is issued; or,

[0128] b) If the maximum offset in the offset sequence increases by a set value and the maximum offset in the first consecutive offset subsequence does not change, an alarm is issued; or,

[0129] c) If the maximum offset in the first continuous offset quantum sequence does not change, and the difference between the maximum offset in the offset sequence and the maximum offset in the first continuous offset quantum sequence reaches a set threshold, an alarm is issued.

[0130] For example, for the corresponding message queue, the offset sequence obtained is 1, 2, 3, 5, 6. After recording the message processing record corresponding to the third message, if the maximum offset in the offset sequence changes within the set time, for example, from 6 to 10, it means that the message in the message queue is being processed, but the offset corresponding to the fourth message has not yet been obtained. In other words, the maximum offset in the first continuous offset subsequence is still 3, which may be due to a processing thread failure, etc., and an alarm is required. Alternatively, the maximum offset in the offset sequence increases by a set value, for example, by 50, from 6 to 56, and the maximum offset in the first continuous offset subsequence is still 3, which may also be due to a processing thread failure, etc., and an alarm is required. Alternatively, the maximum offset in the first continuous offset subsequence is still 3, and the difference between the maximum offset in the offset sequence and 3 reaches the set threshold, for example, the set threshold is 50, and the maximum offset in the offset sequence becomes 53, which may also be due to a processing thread failure, etc., and an alarm is required to promptly discover the existing fault.

[0131] Specifically, the set time, set value and set threshold can all be set as needed, and the embodiments of the present application are not limited to this.

[0132] In practical applications, the message processing method of the embodiment of the present application can be applied to a message processing system based on Kafka. The message processing method of the message processing system based on Kafka is introduced below.

[0133] The message processing system based on Kafka includes a message production end (which can be a server), a Kafka cluster composed of multiple Broker servers, and a consumer end, wherein the consumer end can be the above-mentioned processing server. The message production end can obtain a large number of messages from the message source, and the obtained messages are partitioned and stored on different Broker servers according to a certain key (i.e., the identifier of the message) and an algorithm (such as a hash algorithm), and messages with the same key are stored in the same Broker server. Specifically, a queue shard, i.e., the message queue in the above embodiment, can be deployed on a Broker server to store messages. The consumer end can read messages from multiple queue shards for consumption. For messages in the queue shards, the Kafka message log uses an offset to mark the position of the read message, that is, the offset is the offset of the message consumed in the queue shard, and the consumer end uses the offset offset to determine the starting position of the message to be read next time in the queue shard.

[0134] Figure 6 The detailed flow chart of the message processing method of the message processing system based on Kafka is shown. Figure 6 As shown, the message processing method includes the following steps:

[0135] Step S601: the message production end stores a large number of messages obtained from the message source into multiple queue shards respectively.

[0136] For example Figure 7 As shown, the message production end stores a large number of messages obtained from the message source into queue shard 1, queue shard 2, ..., queue shard n, where n is an integer greater than 1. Specifically, when a large number of messages include messages with the same identifier, the messages with the same identifier are stored in the same queue shard, so that the order of the messages with the same identifier is guaranteed later.

[0137] In step S602, the consumer end continuously pulls each message from multiple queue shards through a pulling thread, and saves each message pulled each time into multiple local queues respectively; wherein, messages with the same identifier in each message are saved in the same local queue, and messages with the same identifier in a local queue are saved in the order in which the messages with the same identifier are arranged in the corresponding queue shards.

[0138] Specifically, the consumer can pull messages from multiple queue shards in batches through a pull thread. When pulling a batch of messages, the pull thread can pull from one queue shard or from multiple queue shards. The following takes pulling messages from one queue shard as an example. The consumer saves each pulled message to the following places: Figure 5 In the local queue 1, local queue 2, ... local queue m shown, m is an integer greater than 1, and the value of m may be the same as or different from the value of n.

[0139] In practical applications, messages with the same identifier may be messages of the same type. Thus, when the messages of the same type are ordered messages, the ordered messages can maintain their sequence in the local queue.

[0140] In order to improve throughput, the consumer side can use multiple processing threads for message processing. When using multiple processing threads, a main thread usually reads a batch of messages and performs grouping operations, and then multiple processing threads process the corresponding grouped messages. After each processing thread processes the corresponding grouped messages, it can report the offset of the processed message, which refers to the position of the message in the corresponding queue shard.

[0141] Specifically, the consumer can save each pulled message in a different local queue according to a certain key (i.e., the message identifier) ​​and algorithm (such as a hash algorithm, etc.), so that messages with the same identifier can be saved in the same local queue.

[0142] Step S603: The consumer uses a processing thread to process each message in the corresponding local queue in the order in which the messages are saved, and obtains a message processing record corresponding to each message, wherein the message processing record includes the offset of the message.

[0143] For example Figure 7 As shown, the messages in local queue 1, local queue 2, ..., local queue m are processed by processing thread 1, processing thread 2, ..., processing thread m respectively. One of the processing threads does not need to wait for other processing threads to complete processing, but can continuously process messages, with good concurrency. For each local queue, a processing thread is used to process each message in the local queue in the order in which the messages are saved, so that the ordered messages can be processed in order to ensure their processing order.

[0144] In step S604, the consumer submits the offset of the processed message to the offset management module through each processing thread, and the offset management module sorts the obtained offsets to obtain an offset sequence.

[0145] It should be noted that the above embodiment takes the case where a batch of messages obtained comes from one queue shard as an example. When a batch of messages obtained comes from multiple queue shards, the offset management module can also divide the obtained offsets according to different queue shards. Specifically, it can be divided according to the carried queue shard identifier, and then the offsets of each queue shard are sorted respectively to obtain the offset sequence corresponding to each queue shard.

[0146] In step S605, the consumer end submits the maximum offset in the first continuous offset subsequence in the offset sequence to the recording module for recording through the offset management module.

[0147] In the above message processing method, since each local queue uses a processing thread to process each message in the local queue in the order in which the messages are saved, the ordered messages can be processed in order to ensure the order of processing. In addition, since multiple local queues are processed by a processing thread respectively, multiple processing threads process messages concurrently, and one of the processing threads does not need to wait for other processing threads, but can continuously process messages, and the concurrency is good. Therefore, the above message processing method can ensure the order of processing of ordered messages, and the concurrency is good.

[0148] Furthermore, if Figure 8 As shown in the figure, after processing the message, each processing thread can submit the offset of the corresponding message to the offset management module. The offset management module can sort the obtained offsets, for example, 1, 2, 4, that is, the messages with offsets 1, 2, and 4 have been processed, and the message with offset 3 has not been processed. At this time, the offset management module can submit offset2 to the recording module for recording to prevent the message with offset 3 from being lost after the shutdown and restart.

[0149] Below Figure 9a and Figure 9b Taking this as an example, the following describes how the embodiments of the present application prevent message loss after a system crash and restart.

[0150] like Figure 9aAs shown, process 1 can represent the process before the crash, and process 2 can represent the process after the crash and restart. The offsets of the messages processed by process 1 are 1, 2, and 4. When the corresponding messages are processed, their offsets are submitted to Kafka. In the related art, Kafka will submit the maximum offset in the processed messages to process 2. At this time, after process 2 obtains offset 4, it knows that the messages corresponding to the offsets before offset 4 have been processed, and will continue to process the messages corresponding to the offsets after offset 4. In this way, the message corresponding to offset 3 will be lost.

[0151] like Figure 9b As shown, in Figure 9a An offset management module is added on the basis of . Process 1 submits the offset of the processed message to the offset management module. The offset management module can store each offset it obtains, and then merge (i.e., sort) the stored multiple offsets. For example, the offset sequence obtained in the submission order is 2, 1, and 4. The offset sequence obtained after merging these offsets is 1, 2, and 4. At this time, since the message corresponding to offset3 has not been processed, the offset management module can submit offset2 to process 2. At this time, after process 2 obtains offset2, it knows that the message corresponding to the offset before offset2 has been processed, and will continue to process the message corresponding to the offset after offset2. In this way, the message corresponding to offset3 can be prevented from being lost.

[0152] Furthermore, when process 2 processes the message corresponding to offset 4, since it has already processed the message corresponding to offset 4 and submitted offset 4, it will not process the message corresponding to offset 4 again, thus avoiding repeated processing of messages.

[0153] Furthermore, the offset management module can also clear expired data in the stored offsets. For example, in the above offset sequence 1, 2, and 4, after the offset management module submits offset2 to process 2, offset1 can be cleared. This is because process 2 has learned that the message corresponding to offset1 has been processed and will continue to process the messages corresponding to the offsets after offset2, and no longer needs to store offset1. In this way, the storage space of the offset management module can be released.

[0154] In addition, the offset management module can also perform fault alarms. For example, after submitting offset2 to process 2, if within the set time, the maximum offset in the above offset sequence changes, for example, from 4 to 10, and offset3 has not been obtained at this time, that is, the message corresponding to offset3 has not been processed, it may be that the processing thread has a fault, etc. Alternatively, if the maximum offset in the above offset sequence increases by a set value, for example, increases by 50, from 4 to 54, and offset3 has not been obtained at this time, it may be that the processing thread has a fault, etc., and an alarm needs to be issued to promptly discover the existing fault. Specifically, the set time can be set as needed, and the embodiment of the present application is not limited to this.

[0155] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a message processing device, referring to Fig.10 As shown, the message processing device includes a storage module 101, a processing module 101 and an acquisition module 103; wherein,

[0156] The storage module 101 is used to continuously obtain each message generated by the message source from at least one message queue, and save each obtained message to at least one target queue each time it is obtained; wherein, messages with the same identifier among the messages are saved to the same target queue, and the messages with the same identifier in a target queue are saved according to the arrangement order of the messages with the same identifier in the corresponding message queue;

[0157] The processing module 102 is used to process each message in the corresponding target queue according to the order in which the messages are saved, using a processing thread for each target queue in the at least one target queue, and obtain a message processing record corresponding to each message;

[0158] The acquisition module 103 is used to obtain the message processing result based on each message processing record obtained.

[0159] Specifically, the storage module obtains messages from multiple message queues in batches, and each time a message is obtained, each message obtained is saved in multiple target queues respectively, and the messages with the same identifier in each message are saved in the same target queue according to the arrangement order in the corresponding message queue. In this way, when the messages with the same identifier are ordered messages, the ordered messages can maintain their order. For each target queue, the processing module uses a processing thread to process each message in the target queue according to the order in which the messages are saved, so that the ordered messages can be processed in order to ensure their processing order. In addition, since multiple target queues are processed by a processing thread respectively, and multiple processing threads process messages concurrently, one of the processing threads does not need to wait for other processing threads, but can continuously process messages, and the concurrency is good.

[0160] In a possible embodiment, each message obtained from at least one message queue carries a message queue identifier, and the device further includes:

[0161] A determination module, for determining, for each message processing record corresponding to each currently processed message, the message processing record of each message from the corresponding message queue according to the message queue identifier carried by each processed message;

[0162] The sorting module is used to arrange the message processing records of each message in the corresponding message queue according to the order of each message in the message queue to obtain the arranged message processing records corresponding to the corresponding message queue.

[0163] In a possible embodiment, the message processing record includes an offset of a corresponding message, and the offsets of each message obtained from the same message queue are continuously increased, and the device further includes:

[0164] A determination module, configured to determine an offset sequence for a corresponding message queue according to an offset in an arranged message processing record corresponding to the message queue;

[0165] The recording module is used to determine the maximum offset in the first continuous offset subsequence in the offset sequence, and record the message corresponding to the maximum offset that has been processed in the corresponding message queue.

[0166] In a possible embodiment, the device may further include:

[0167] The deletion module is used to delete the message processing records corresponding to the offsets except the maximum offset in the first continuous offset subsequence.

[0168] In a possible embodiment, the device may further include:

[0169] an alarm module, for generating an alarm if the maximum offset in the offset sequence changes within a set time and the maximum offset in the first consecutive offset subsequence does not change; or,

[0170] If the maximum offset in the offset sequence increases by the set value and the maximum offset in the first consecutive offset subsequence does not change, an alarm is issued; or,

[0171] If the maximum offset in the first consecutive offset subsequence does not change, and the difference between the maximum offset in the offset sequence and the maximum offset in the first consecutive offset subsequence reaches a set threshold, an alarm is issued.

[0172] In a possible embodiment, the storage module 101 may be specifically used for:

[0173] Perform hash operations on the identifiers of each message to obtain corresponding hash values;

[0174] Save each message to the target queue associated with the corresponding hash value.

[0175] Specifically, different identifiers can be represented by different keys. For example, for a message containing key1, key1 can be hashed to obtain a corresponding hash value. Each target queue can be pre-set to be associated with a corresponding hash value. For example, the hash value associated with target queue 1 is H1, the hash value associated with target queue 2 is H2, and the hash value associated with target queue 3 is H3. When the hash value corresponding to the above key1 is H1, all messages containing key1 are saved in target queue 1. Thus, messages with the same identifier can be saved in the same target queue.

[0176] Based on the same inventive concept as the above method embodiment and device embodiment, the present application embodiment also provides an electronic device. The electronic device may be a server, such as Figure 4 The processing server 300 shown in FIG. 1 is a schematic diagram of a processing server 300. In this embodiment, the structure of the electronic device can be as follows: Fig.11 As shown, it includes a memory 301 , a communication module 303 and one or more processors 302 .

[0177] The memory 301 is used to store computer programs executed by the processor 302. The memory 301 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0178] The processor 302 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 302 is configured to implement the above message processing method when calling the computer program stored in the memory 301 .

[0179] The communication module 303 is used to communicate with the terminal device and obtain voice data.

[0180] The specific connection medium between the memory 301, the communication module 303 and the processor 302 is not limited in the embodiment of the present application. Fig.11 In the embodiment, the memory 301 and the processor 302 are connected via a bus 304. The bus 304 is Fig.11 The connections between other components are shown in bold lines, and are not intended to be limiting. Bus 304 can be divided into address bus, data bus, control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0181] The embodiment of the present application also provides a computer storage medium, the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the message processing method in the above embodiment.

[0182] In some possible implementations, various aspects of the method for determining multimedia content provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the message processing method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 5 The flow of message processing in steps S501 to S503 is shown.

[0183] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0184] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0186] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0187] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0188] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A message processing method, characterized in that: include: Continuously obtain messages generated by a message source from at least one message queue, and save the obtained multiple messages into at least one target queue each time each time the messages are obtained; wherein, among the multiple messages, messages with the same identifier from the same message queue are saved into the same target queue, and the messages with the same identifier in a target queue are saved in the order in which the messages with the same identifier are arranged in the corresponding message queue, and the messages with the same identifier include ordered messages, and the ordered messages represent messages that are processed in the order in which they are obtained; For each target queue in the at least one target queue, using a processing thread to process each message in the corresponding target queue in the order in which the messages are saved, and obtaining a message processing record corresponding to each message; Based on each obtained message processing record, a message processing result is obtained.

2. The method according to claim 1, characterized in that Each message obtained from at least one message queue carries a message queue identifier, and the method further includes: For the message processing records corresponding to each currently processed message, determine the message processing records of each message from the corresponding message queue according to the message queue identifier carried by each processed message; The message processing records of each message in the corresponding message queue are arranged according to the order of each message in the message queue to obtain the arranged message processing records corresponding to the corresponding message queue.

3. The method according to claim 2, characterized in that The message processing record includes the offset of the corresponding message in the corresponding message queue, and the offset of each message obtained from the same message queue increases continuously, then the method further includes: For a corresponding message queue, determining an offset sequence according to an offset in the arranged message processing record corresponding to the message queue; The maximum offset in the first continuous offset subsequence in the offset sequence is determined, and the message corresponding to the maximum offset that has been processed in the corresponding message queue is recorded.

4. The method according to claim 3, characterized in that After the message queue corresponding to the record has processed the message corresponding to the maximum offset, the method further includes: The message processing records corresponding to the offsets other than the maximum offset in the first continuous offset subsequence are deleted.

5. The method according to claim 3, characterized in that: After the message queue corresponding to the record has processed the message corresponding to the maximum offset, the method further includes: If within the set time, the maximum offset in the offset sequence changes, and the maximum offset in the first consecutive offset subsequence does not change, an alarm is issued; or, If the maximum offset in the offset sequence increases by a set value and the maximum offset in the first continuous offset subsequence does not change, an alarm is issued; or, If the maximum offset in the first continuous offset quantum sequence does not change, and the difference between the maximum offset in the offset sequence and the maximum offset in the first continuous offset quantum sequence reaches a set threshold, an alarm is issued.

6. The method according to any one of claims 1 to 5, characterized in that: The step of saving each acquired message to at least one target queue includes: Performing hash operations on the identifiers of the respective messages to obtain corresponding hash values; Each of the messages is saved in a target queue associated with a corresponding hash value.

7. A message processing device, characterized in that: include: A saving module, used for continuously acquiring messages generated by a message source from at least one message queue, and each time acquiring, saving the acquired multiple messages to at least one target queue; wherein, among the multiple messages, messages with the same identifier from the same message queue are saved to the same target queue, and the messages with the same identifier in a target queue are saved according to the arrangement order of the messages with the same identifier in the corresponding message queue, and the messages with the same identifier include ordered messages, and the ordered messages represent messages processed in the order of acquisition; A processing module, configured to process each message in the corresponding target queue using a processing thread according to the order in which the messages are saved, for each target queue in the at least one target queue, and obtain a message processing record corresponding to each message; The acquisition module is used to obtain the message processing result based on each message processing record obtained.

8. The device according to claim 7, characterized in that Each message obtained from at least one message queue carries a message queue identifier, and the device further includes: A determination module, for determining, for each message processing record corresponding to each currently processed message, the message processing record of each message from the corresponding message queue according to the message queue identifier carried by each processed message; The sorting module is used to arrange the message processing records of each message in the corresponding message queue according to the order of each message in the message queue to obtain the arranged message processing records corresponding to the corresponding message queue.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: It includes program codes. When the program codes are run on an electronic device, the program codes are used to enable the electronic device to execute the steps of the method according to any one of claims 1 to 6.

Images