Message processing method and device, related equipment, storage medium and computer program product

Abstract

The invention discloses a message processing method and device, related equipment, a storage medium and a computer program product. The method comprises the steps that a first device receives a first message set sent by a second device, and the first message set comprises at least two messages; the first message set is written into a data log, a first index associated with the first message set is generated, and the first index is used for pulling messages from the data log. According to the technical scheme provided by the invention, the server takes the batch messages as a whole and generates the consumption index corresponding to the batch messages, so that the messages in the log file can be quickly searched during message consumption, resource consumption caused by message splitting can be avoided, the message processing efficiency is improved, and meanwhile, the message processing efficiency is improved. And the complexity of message consumption is reduced, so that the scheme can be suitable for a message scene of stream processing.

Description

Technical Field

[0001] The present application relates to the field of network transmission technology, and in particular to a message processing method, apparatus, related equipment, storage medium and computer program product. Background Art

[0002] Rocket Message Queue (MQ) is a high-performance, high-reliability distributed message queue that supports multiple sending modes, including synchronous sending, asynchronous sending, and one-way sending. It also supports batch processing of messages (including batch sending and batch consumption).

[0003] Compared with sending a single message, batch processing of messages can effectively improve the throughput of Rocket MQ. However, there is a problem of high resource usage in this process. Summary of the invention

[0004] To solve related technical problems, the embodiments of the present application provide a message processing method, apparatus, related equipment, storage medium and computer program product.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The embodiment of the present application provides a message processing method, which is applied to a first device, including:

[0007] Receiving a first message set sent by a second device, where the first message set includes at least two messages;

[0008] The first message set is written into a data log, and a first index associated with the first message set is generated, where the first index is used to pull messages from the data log.

[0009] In the above solution, the first index includes one or more of the following:

[0010] first information, where the first information is used to indicate a position of the first message set in the data log;

[0011] second information, wherein the second information represents the number of bytes of the first message set;

[0012] third information, where the third information is used to indicate a label of the first message set;

[0013] fourth information, where the fourth information is used to indicate a position of the first index in the index file;

[0014] The fifth information is used to indicate the number of messages in the first message set.

[0015] In the above scheme, the method further comprises:

[0016] receiving sixth information sent by a third device, the sixth information being used to request to pull a message to be consumed, the sixth information comprising location-related information associated with the message to be consumed;

[0017] Determine seventh information using the location-related information and the first index, where the seventh information is used to indicate a position of the message to be consumed in the data log;

[0018] Using the seventh information, pulling the message to be consumed from the data log;

[0019] The message to be consumed is sent to the third device.

[0020] In the above solution, the determining the seventh information by using the site-related information and the first index includes:

[0021] The seventh information is obtained by searching the first index using the location-related information in a binary search manner.

[0022] In the above solution, the receiving of the first message set sent by the second device includes:

[0023] In a case where the first message set meets a sending condition, the first message set sent by the second device is received, where the sending condition is associated with a message size and / or message accumulation time of the first message set.

[0024] The embodiment of the present application also provides a message processing method, which is applied to a second device, including:

[0025] Generate a first message set, the first message set including at least two messages, the first message set being used to generate a first index associated with the first message set, the first index being used to pull messages from a data log;

[0026] The first message set is sent to a first device.

[0027] In the above solution, the first index includes one or more of the following:

[0028] first information, the first information being used to indicate a position of the first message set in the data log;

[0029] second information, wherein the second information represents the number of bytes of the first message set;

[0030] third information, where the third information is used to indicate a label of the first message set;

[0031] fourth information, where the fourth information is used to indicate a position of the first index in the index file;

[0032] The fifth information is used to indicate the number of messages in the first message set.

[0033] In the above solution, the sending the first message set to the first device includes:

[0034] In a case where the first message set meets a sending condition, the first message set is sent to the first device, where the sending condition is associated with a message size and / or message accumulation time of the first message set.

[0035] In the above solution, generating the first message set includes:

[0036] Writing the at least two messages in sequence through multiple threads to obtain the first message set, each thread being used to write one message;

[0037] Determine whether the first message set meets the sending condition.

[0038] In the above solution, the multiple threads include a first thread, the first thread is at least used to write a first message, and the step of writing the at least two messages in sequence through the multiple threads to obtain the first message set includes:

[0039] Setting lock information for the first message set through the first thread, where the lock information is used to lock the first message set;

[0040] The first message is written by the first thread.

[0041] The embodiment of the present application further provides a first device, comprising: a first processor and a first memory for storing a computer program that can be run on the processor,

[0042] Wherein, when the first processor is used to run the computer program, it executes the steps of any one of the above-mentioned methods on the first device side.

[0043] The embodiment of the present application further provides a second device, comprising: a second processor and a second memory for storing a computer program that can be run on the processor,

[0044] Wherein, the second processor is used to execute the steps of any one of the above-mentioned methods on the second device side when running the computer program.

[0045] An embodiment of the present application also provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the steps of any of the methods on the first device side described above, or implements the steps of any of the methods on the second device side described above.

[0046] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods on the first device side, or implements the steps of any of the above-mentioned methods on the second device side.

[0047] The message processing method, apparatus, related equipment, storage medium and computer program product provided by the embodiments of the present application, the first device receives a first message set sent by the second device, the first message set includes at least two messages; the first message set is written into a data log, and a first index associated with the first message set is generated, the first index is used to pull messages from the data log. In the technical solution provided by the present application, during the batch message transmission process, the server directly saves the batch messages and constructs an index of the batch messages, so that the message data in the log file can be quickly found through the index in the message consumption scenario; in the above process, since the batch messages are indexed as a whole, the central processing unit (CPU) and disk loss caused by splitting the batch messages can be reduced, thereby improving the message processing performance, and at the same time, improving the throughput of the message queue. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a batch message processing flow in the related art;

[0049] Figure 2 This is a flow chart of the first message processing method according to an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a message consumption process according to an embodiment of the present application;

[0051] Figure 4 This is a flow chart of the second message processing method according to an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a message accumulation process according to an embodiment of the present application;

[0053] Figure 6 This is a schematic diagram of the structure of the first message processing device in the embodiment of the present application;

[0054] Figure 7 This is a schematic diagram of the structure of the second message processing device according to an embodiment of the present application;

[0055] Figure 8 This is a schematic diagram of the first device structure of the embodiment of the present application;

[0056] Fig. 9 This is a schematic diagram of the second device structure of the embodiment of the present application;

[0057] Fig.10 This is a schematic diagram of the message processing system structure of an embodiment of the present application. DETAILED DESCRIPTION

[0058] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0059] In the related Rocket MQ message batch processing solution, such as Figure 1 As shown, when sending batch messages, the sender (Client in English) (also known as the message producer) manually groups a batch of messages into a collection and calls the sending interface to send them; and the server (Broker in English) receives the batch messages through the Send Message Processor (Send Message Processor in English), parses each message from the collection (also known as splitting the batch messages) and writes it into the data log (CommitLog in English). At the same time, the corresponding message index (such as the consumption index (ConsumQueue in English) and IndexFile) is generated through the background thread.

[0060] When consuming batch messages, the client sends a pull request (PullMessage in English) to the server to request to pull the latest message transmitted to the server. Since each client is assigned a MessageQueue, the server uses the pull request processor (Pull Message Processor in English) to determine whether the message should be pulled to the client's local cache based on the message index (specifically ConsumeQueue) and the consumption location information. When the consumption location information in the pull request is the same as the write location information in the current ConsumeQueue, it means that the latest message transmitted to the server has been provided to the client. In this case, the server will not pull the message; otherwise, the server will find the latest message in the corresponding ConsumeQueue from the CommitLog based on the write location information in the consumption index ConsumeQueue and return it to the client, so that the client can store the pulled message in the local cache, and then call the consumer thread to consume the message.

[0061] In the above scheme, on the one hand, the message collection needs to be manually passed in when sending a message, and the size of the message collection depends entirely on the experience of the sender. If the message collection is too large, some messages will wait too long to be sent, which may cause delays in business calls; if the message collection is too small, the number of calls to the message production application programming interface (API) will be similar to that of a single message, and the purpose of batch sending cannot be achieved. On the other hand, after receiving the message collection, the server needs to decode the batch messages to write each message in the collection into the CommitLog in sequence. Then, for each message, the background thread will continuously build the ConsumeQueue and IndexFile, which will result in excess CPU and disk usage.

[0062] Based on this, in various embodiments of the present application, a new batch message processing mechanism is designed at the same time, so that the server can write the batch message as a whole when writing the message, and at the same time, the asynchronous thread builds the consumption index of the batch message. Since the batch message is directly saved and the index is only built once, it can reduce the CPU and disk usage of the server, improve the efficiency of client message processing, and reduce the complexity of client consumption, which is more suitable for the message scenario of stream processing.

[0063] The present application embodiment provides a message processing method, such as Figure 2 As shown, applied to a first device, the method includes:

[0064] Step 201: Receive a first message set sent by a second device, where the first message set includes at least two messages;

[0065] Step 202: Write the first message set into a data log, and generate a first index associated with the first message set, where the first index is used to pull messages from the data log.

[0066] In actual application, the first device can be called a server, a service device, etc., and correspondingly, the second device can be called a message producer, a sender, a sending device, etc. The embodiment of the present application does not limit the names of the first device and the second device, as long as their functions are realized.

[0067] In addition, the first message set can be understood as a message set formed by accumulating multiple messages, and therefore, the messages included in the first message set can be called batch messages; wherein, the first message set can be stored in a first message queue, specifically in Rocket MQ.

[0068] In actual application, in step 201, the second device can automatically accumulate messages and provide the first message set to the first device when the set sending conditions are met. In this way, messages can be quickly sent under different network conditions and inconsistent message sizes, thereby improving the overall throughput of the transmission system.

[0069] Specifically, in one embodiment, the receiving a first message set sent by the second device includes:

[0070] In a case where the first message set meets a sending condition, the first message set sent by the second device is received, where the sending condition is associated with a message size and / or message accumulation time of the first message set.

[0071] Among them, in actual application, when the sending condition is associated with the message size of the first message set, the sending condition may include that the size of the accumulated messages in the first message set reaches a preset first threshold, and the value of the first threshold can be set as needed, which is not limited to the embodiment of the present application; when the sending condition is associated with the message accumulation time of the first message set, the sending condition may include that the message accumulation time in the first message set (which can also be understood as waiting time) reaches a preset first duration (also called maximum batch accumulation time), and the value of the first duration can be set as needed, which is not limited to the embodiment of the present application.

[0072] In actual application, after receiving the first message set, the first device can send eighth information to the second device, and the eighth information indicates that the first message set has been successfully received. The eighth information can be understood as the sending result of the first message set.

[0073] Next, the first device may write the multiple messages included in the first message set as a whole into the data log (which may be expressed as Commit Log in English) (which may also be called a log file), and at the same time, construct the first index through a thread (which may also be called an asynchronous thread); wherein the first index may be called a batch consumption index (which may be expressed as Batch Consume Queue in English), which is at least used to pull messages from the data log in a message consumption scenario for consumers to consume.

[0074] In one embodiment, the first index may include one or more of the following (also understood as at least one or at least one):

[0075] first information, the first information being used to indicate a position of the first message set in the data log;

[0076] second information, wherein the second information represents the number of bytes of the first message set;

[0077] third information, where the third information is used to indicate a label of the first message set;

[0078] fourth information, where the fourth information is used to indicate a position of the first index in the index file;

[0079] The fifth information is used to indicate the number of messages in the first message set.

[0080] Among them, in actual application, the first information can be called a physical offset (expressed as PhysicalOffset in English), and the size of the first information can be 8 bytes; the second information can be called a message size (expressed as Body Size in English), which can reflect the size of the message body of the first message set, and the size of the second information can be 4 bytes; the third information can be called a tag value (expressed as Tag Hash in English), which can reflect the hash value of the tag of the message in the first message set, so as to filter the message in the message scenario, and the size of the third information can be 8 bytes; the fourth information can be called a queue offset (expressed as Queue Offset in English). Since the index file may contain indexes corresponding to multiple message sets, the fourth information can reflect the logical position of the first index corresponding to the first message set in the index file, and the size of the fourth information can be 4 bytes; the fifth information can be called a batch size (expressed as Batch Size in English). Since the first message set contains multiple messages, the fifth information can reflect the number of batch messages, so as to count the number of messages. In addition, the first index may also include reserved information (which may be expressed as Reserve in English) for extended purposes, and the size of the reserved information may be 8 bytes.

[0081] In actual application, after constructing the first index, in a consumption scenario, the first device can pull corresponding messages for a consumer (which can be expressed as Consumer in English) through the first index.

[0082] Based on this, in one embodiment, the method may further include:

[0083] receiving sixth information sent by a third device, the sixth information being used to request to pull a message to be consumed, the sixth information comprising location-related information associated with the message to be consumed;

[0084] Determine seventh information using the location-related information and the first index, where the seventh information is used to indicate a position of the message to be consumed in the data log;

[0085] Using the seventh information, pulling the message to be consumed from the data log;

[0086] The message to be consumed is sent to the third device.

[0087] In actual application, the third device can be understood as a client associated with the consumer, that is, the consumer can send the sixth information (which can also be understood as a pull request) through the third device; wherein the location-related information is used to describe the number and / or location of the messages to be consumed, such as 32 messages with a starting location of 100.

[0088] In actual application, in the process of determining the seventh information, the first device may use binary search for the message to be consumed to improve the message search efficiency.

[0089] Specifically, in one embodiment, the determining the seventh information by using the location-related information and the first index includes:

[0090] The seventh information is obtained by searching the first index using the location-related information in a binary search manner.

[0091] Here, using the location-related information, the first device can perform a binary search to obtain the specific location of the message to be consumed in the data log, thereby avoiding traversing the data log and improving the efficiency of message search.

[0092] Exemplarily, assuming that the index file contains indexes corresponding to N message sets (N is an integer greater than or equal to 2), the first device can determine the first index associated with the location-related information from the index file, and then search for the seventh information from the first index.

[0093] In actual application, when the message to be consumed contains multiple messages, the third device can decompose the message to be consumed (which can also be understood as an unpacking operation), consume the decomposed message to be consumed, and then call the consumption logic to send the ninth information to the first device, and the ninth information represents the consumption result of the message to be consumed (such as consumption success or consumption failure). When the message to be consumed contains one message, the third device can directly consume the message to be consumed, and then send the ninth information to the first device.

[0094] For example, Figure 3As shown, the first device writes different message sets to the data log, and at the same time, constructs an index file through an asynchronous thread, the index file at least contains the second index (corresponding to index=0) and the first index (corresponding to index=2) corresponding to other message sets, and the first index contains the first information, the second information, the third information, the fourth information and the fifth information. In the scenario where the consumer has consumption needs, the first device determines the corresponding message to be consumed from the data log based on the index file and the location-related information, and sends the determined message to be consumed to the third device, so that the third device can consume the message.

[0095] In the embodiment of the present application, the newly designed index structure BatchConsumeQueue can quickly find the message data in the data log in the consumption scenario. At the same time, since some information of the batch message itself (such as the batch size, etc.) is saved, it is also convenient for consumers to quickly count some statistical data. In addition, since the batch message is saved and indexed as a whole, the CPU and disk loss caused by splitting the batch message is avoided, which improves the processing performance of the second device to a certain extent, that is, reduces the pressure on the second device and improves the efficiency of message pulling.

[0096] Accordingly, the embodiment of the present application also provides a message processing method, which is applied to a second device, such as Figure 4 As shown, the following steps are included:

[0097] Step 401: Generate a first message set, the first message set includes at least two messages, the first message set is used to generate a first index associated with the first message set, and the first index is used to pull messages from a data log;

[0098] Step 402: Send the first message set to the first device.

[0099] In actual application, for the first message set, the second device may accumulate messages through multiple threads concurrently to generate the first message set.

[0100] Specifically, in one embodiment, generating the first message set includes:

[0101] Writing the at least two messages in sequence through multiple threads to obtain the first message set, each thread being used to write one message;

[0102] Determine whether the first message set meets the sending condition.

[0103] In actual application, during the process of writing a message, the second device may set a lock competition mechanism to ensure the correctness of the shared data.

[0104] Specifically, in one embodiment, the multiple threads include a first thread, the first thread is at least used to write a first message, and the step of writing the at least two messages in sequence through the multiple threads to obtain the first message set includes:

[0105] Setting lock information for the first message set through the first thread, where the lock information is used to lock the first message set;

[0106] The first message is written by the first thread.

[0107] The lock information may be obtained from an operating system (such as Mutex). The embodiment of the present application does not limit the method of obtaining the lock information, as long as its function is implemented.

[0108] In actual application, for the first message, the first thread sets lock information for the first message set to lock the first message set; the first thread writes the first message, obtains the index of the first message in the first message set, and then updates the number and size of messages in the current first message set. In this way, it can be ensured that other threads will not write messages synchronously when writing the first message, and the orderly writing of messages is realized, while the accuracy of the number and size of messages is guaranteed.

[0109] Here, after writing the first message, the second device can determine whether the current first message set meets the sending condition; if the first message set does not meet the sending condition, the second device can adjust the state of the first thread from a working state to a waiting state, and continue to write messages through the next thread (such as the second thread) until the first message set meets the sending condition; wherein the sending condition can be associated with the message size and / or message accumulation time of the first message set.

[0110] In actual application, when the sending condition includes that the size of messages accumulated in the first message set reaches the first threshold, if the first threshold is set to a large value, it may cause all threads to be in a waiting state, thereby causing a delay in the sending of the first message set. Therefore, the second device may also set the sending condition to be associated with the message accumulation time of the first message set, that is, the sending condition may also include that the time for accumulation of messages in the first message set reaches a first duration. In this way, the second device may also determine whether the first message set meets the sending condition based on the message accumulation time.

[0111] In actual application, when the first message set meets the sending condition, the second device can directly send the first message set.

[0112] Specifically, in one embodiment, sending the first message set to the first device includes:

[0113] In a case where the first message set meets a sending condition, the first message set is sent to the first device, where the sending condition is associated with a message size and / or message accumulation time of the first message set.

[0114] Then, the second device can receive the eighth information to learn that the first device has successfully received the first message set. At the same time, the second device can wake up all threads associated with the first message set that are in a waiting state, and split the eighth information according to the index of the written message in the first message set, so that each thread can obtain the sending result corresponding to the message written by itself.

[0115] In the embodiments of the present application, by reasonably designing the lock competition in a multi-threaded environment, the automatic batching and sending of messages under the same topic (which can be expressed as Topic in English) is realized. In this way, the problem of slow sending or excessive sending frequency and low actual throughput caused by manually setting the message collection can be avoided. At the same time, it can also achieve fast message sending under different network conditions and different message sizes, thereby improving the overall throughput of the system.

[0116] It should be noted that, in the process of accumulating messages, the second device can determine whether each message can be sent in batches according to the type of each message; if the type of the message does not meet the requirements (such as messages with high latency requirements), it means that the message cannot be sent in batches. In this case, the second device can directly send the message; if the type of the message meets the requirements, it means that the message can be sent in batches. In this case, the second device can write the message through multiple threads. In this way, messages with high latency requirements (such as delayed messages, transaction messages, etc.) can be sent in real time;

[0117] For example, Figure 5As shown, it is assumed that an automatic batch accumulator (which can be expressed as accumulator in English) (also called automatic batch accumulator) is set on the second device. When message 1 is received, thread 1 (thread1) determines that message 1 can be sent in batches, and writes message 1 into the batch list of the automatic batch accumulator. Since the current message size and waiting time accumulated by the automatic batch accumulator do not meet the sending conditions, the automatic batch accumulator adjusts the state of thread 1 to the waiting state. In this process, the thread can be monitored (which can be expressed as guard thread in English) and the thread can be notified (which can be expressed as notify in English) in time. The automatic batch sender adjusts the state of thread 2 to the waiting state; when message 3 is received, thread 3 (thread3) determines that message 3 can be sent in batches, and writes message 2 into the batch list. Since the message size and waiting time accumulated by the current automatic batch sender do not meet the sending conditions, the automatic batch sender adjusts the state of thread 2 to the waiting state; when message 3 is received, thread 3 (thread3) determines that message 3 can be sent in batches, and writes message 3 into the batch list. Since the message size and waiting time accumulated by the current automatic batch sender meet the sending conditions, the automatic batch sender sends an instruction to thread 3, so that thread 3 calls a batch message sending request (which can be expressed as send call in English) to send the accumulated messages (i.e., the first message set) to the first device. Then, thread 3 feeds back the sending result to the automatic batch sender, so that the automatic batch sender can notify all threads that the batch messages have been successfully sent (which can be expressed as notify all in English).

[0118] The message processing method provided in the embodiment of the present application is that a first device receives a first message set sent by a second device, and the first message set includes at least two messages; the first message set is written into a data log, and a first index associated with the first message set is generated, and the first index is used to pull messages from the data log. In the technical solution provided by the present application, during the batch message transmission process, the server directly saves the batch messages and builds an index for the batch messages, so that the message data in the log file can be quickly found through the index in the message consumption scenario; in the above process, since the batch messages are indexed as a whole, it is possible to reduce the CPU and disk loss caused by splitting the batch messages, thereby improving the message processing performance, and at the same time, improving the throughput of the message queue.

[0119] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a message processing device, which is set on the first device, such as Figure 6 As shown, the device comprises:

[0120] A receiving unit 601 is configured to receive a first message set sent by a second device, where the first message set includes at least two messages;

[0121] The writing unit 602 is used to write the first message set into a data log and generate a first index associated with the first message set, where the first index is used to pull messages from the data log.

[0122] In one embodiment, the device may further include: a transmission unit; wherein,

[0123] The receiving unit 601 is further configured to receive sixth information sent by a third device, wherein the sixth information is used to request to pull a message to be consumed, and the sixth information includes location-related information associated with the message to be consumed;

[0124] The writing unit 602 is further used to determine seventh information using the location-related information and the first index, where the seventh information is used to indicate the position of the message to be consumed in the data log; and to pull the message to be consumed from the data log using the seventh information;

[0125] The transmission unit is used to send the message to be consumed to the third device.

[0126] In one embodiment, the writing unit 602 is used to use the location-related information to search for the seventh information from the first index by means of a binary search.

[0127] In one embodiment, the receiving unit 601 is used to receive the first message set sent by the second device when the first message set meets a sending condition, and the sending condition is associated with the message size and / or message accumulation time of the first message set.

[0128] In actual application, the receiving unit 601 and the transmitting unit may be implemented by a communication interface in a message processing device; and the writing unit 602 may be implemented by a processor in the message processing device.

[0129] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a message processing device, which is set on the second device, such as Figure 7 As shown, the device comprises:

[0130] A generating unit 701 is configured to generate a first message set, wherein the first message set includes at least two messages, and the first message set is used to generate a first index associated with the first message set, and the first index is used to pull messages from a data log;

[0131] The sending unit 702 is configured to send the first message set to a first device.

[0132] In one embodiment, the sending unit 702 is used to send the first message set to the first device when the first message set meets a sending condition, and the sending condition is associated with the message size and / or message accumulation time of the first message set.

[0133] In one embodiment, the generating unit 701 is used to:

[0134] Writing the at least two messages in sequence through multiple threads to obtain the first message set, each thread being used to write one message;

[0135] Determine whether the first message set meets the sending condition.

[0136] In one embodiment, the multiple threads include a first thread, the first thread is at least used to write a first message, and the generating unit 701 is used to:

[0137] Setting lock information for the first message set through the first thread, where the lock information is used to lock the first message set;

[0138] The first message is written by the first thread.

[0139] In actual application, the generating unit 701 may be implemented by a processor in a message processing device; and the sending unit 702 may be implemented by a communication interface in the message processing device.

[0140] It should be noted that: when the message processing device provided in the above embodiment performs message processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the message processing device provided in the above embodiment and the message processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0141] Based on the hardware implementation of the above program module, and in order to implement the method on the first device side of the embodiment of the present application, the embodiment of the present application also provides a first device, such as Figure 8 As shown, the first device 800 includes:

[0142] The first communication interface 801 is capable of exchanging information with the second device;

[0143] A first processor 802 is connected to the first communication interface 801 to implement information interaction with the second device, and is used to execute the method provided by one or more technical solutions on the first device side when running a computer program;

[0144] A first memory 803 , in which the computer program is stored.

[0145] Specifically, the first communication interface 801 is used to receive a first message set sent by a second device, where the first message set includes at least two messages;

[0146] The first processor 802 is used to write the first message set into a data log and generate a first index associated with the first message set, where the first index is used to pull messages from the data log.

[0147] In one embodiment, the first communication interface 801 is further used to receive sixth information sent by a third device, the sixth information is used to request to pull the message to be consumed, and the sixth information includes location-related information associated with the message to be consumed;

[0148] The first processor 802 is also used to use the location-related information and the first index to determine seventh information, where the seventh information is used to indicate the position of the message to be consumed in the data log; use the seventh information to pull the message to be consumed from the data log; and send the message to be consumed to the third device through the first communication interface 801.

[0149] In one embodiment, the first communication interface 801 is used to search for the seventh information from the first index using the location-related information in a binary search manner.

[0150] In one embodiment, the first communication interface 801 is used to receive the first message set sent by the second device when the first message set meets a sending condition, and the sending condition is associated with the message size and / or message accumulation time of the first message set.

[0151] It should be noted that the specific processing process of the first communication interface 801 and the first processor 802 can be understood by referring to the above method.

[0152] Of course, in actual application, the various components in the first device 800 are coupled together through the bus system 804. It can be understood that the bus system 804 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8Various buses are labeled as bus system 804 .

[0153] The first memory 803 in the embodiment of the present application is used to store various types of data to support the operation of the first device 800. Examples of such data include: any computer program used to operate on the first device 800.

[0154] The method disclosed in the above embodiment of the present application can be applied to the first processor 802, or implemented by the first processor 802. The first processor 802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the first processor 802 or an instruction in the form of software. The above-mentioned first processor 802 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The first processor 802 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the first memory 803, and the first processor 802 reads the information in the first memory 803 and completes the steps of the above method in combination with its hardware.

[0155] In an exemplary embodiment, the first device 800 can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0156] Based on the hardware implementation of the above program module, and in order to implement the method on the second device side of the embodiment of the present application, the embodiment of the present application also provides a second device, such as Fig. 9 As shown, the second device 900 includes:

[0157] The second communication interface 901 is capable of exchanging information with the first device;

[0158] A second processor 902 is connected to the second communication interface 901 to implement information interaction with the first device, and is used to execute the method provided by one or more technical solutions on the second device side when running a computer program;

[0159] A second memory 903 , in which the computer program is stored.

[0160] Specifically, the second processor 902 is used to generate a first message set, the first message set includes at least two messages, the first message set is used to generate a first index associated with the first message set, and the first index is used to pull messages from the data log;

[0161] The second communication interface 901 is used to send the first message set to the first device.

[0162] In one embodiment, the second communication interface 901 is used to send the first message set to the first device when the first message set meets a sending condition, and the sending condition is associated with the message size and / or message accumulation time of the first message set.

[0163] In one embodiment, the second processor 902 is configured to:

[0164] Writing the at least two messages in sequence through multiple threads to obtain the first message set, each thread being used to write one message;

[0165] Determine whether the first message set meets the sending condition.

[0166] In one embodiment, the multiple threads include a first thread, the first thread is at least used to write a first message, and the second processor 902 is used to:

[0167] Setting lock information for the first message set by the first thread, wherein the lock information is used to lock the first message set;

[0168] The first message is written by the first thread.

[0169] It should be noted that the specific processing process of the second communication interface 901 and the second processor 902 can be understood by referring to the above method.

[0170] Of course, in actual application, the various components in the second device 900 are coupled together through the bus system 904. It can be understood that the bus system 904 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig. 9 Various buses are labeled as bus system 904.

[0171] The second memory 903 in the embodiment of the present application is used to store various types of data to support the operation of the second device 900. Examples of such data include: any computer program used to operate on the second device 900.

[0172] The method disclosed in the above embodiment of the present application can be applied to the second processor 902, or implemented by the second processor 902. The second processor 902 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the second processor 902 or an instruction in the form of software. The above-mentioned second processor 902 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The second processor 902 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the second memory 903, and the second processor 902 reads the information in the second memory 903 and completes the steps of the above method in combination with its hardware.

[0173] In an exemplary embodiment, the second device 900 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general purpose processors, controllers, MCUs, Microprocessors, or other electronic components to perform the aforementioned methods.

[0174] It can be understood that the memory (first memory 803, second memory 903) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0175] In order to implement the method provided in the embodiment of the present application, the embodiment of the present application also provides a message processing system, such as Fig.10 As shown, the system includes: a first device 1001 and a second device 1002.

[0176] Here, it should be noted that the specific processing procedures of the first device 1001 and the second device 1002 have been described in detail above and will not be repeated here.

[0177] In an exemplary embodiment, the embodiment of the present application further provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory 803 storing a computer program, the computer program can be executed by the first processor 802 of the first device 800 to complete the steps of the first device side method, and for another example, including a second memory 903 storing a computer program, the computer program can be executed by the second processor 902 of the second device 900 to complete the steps of the second device side method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM.

[0178] In an exemplary embodiment, the embodiment of the present application also provides a computer program product, including a computer program, which can be executed by the first processor 802 of the first device 800 to complete the steps described in the aforementioned first device side method, or the computer program can be executed by the second processor 902 of the second device 900 to complete the steps described in the aforementioned second device side method.

[0179] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0180] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0181] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A message processing method, characterized in that: Applied to a first device, comprising: Receiving a first message set sent by a second device, wherein the first message set includes at least two messages; The first message set is written into a data log, and a first index associated with the first message set is generated, where the first index is used to pull messages from the data log.

2. The method according to claim 1, characterized in that The first index includes one or more of the following: first information, the first information being used to indicate a position of the first message set in the data log; second information, wherein the second information represents the number of bytes of the first message set; third information, where the third information is used to indicate a label of the first message set; fourth information, where the fourth information is used to indicate a position of the first index in the index file; The fifth information is used to indicate the number of messages in the first message set.

3. The method according to claim 1, characterized in that The method further comprises: receiving sixth information sent by a third device, the sixth information being used to request to pull a message to be consumed, the sixth information comprising location-related information associated with the message to be consumed; Determine seventh information using the location-related information and the first index, where the seventh information is used to indicate a position of the message to be consumed in the data log; Using the seventh information, pulling the message to be consumed from the data log; The message to be consumed is sent to the third device.

4. The method according to claim 3, characterized in that Determining seventh information by using the location-related information and the first index includes: The seventh information is obtained by searching the first index using the location-related information in a binary search manner.

5. The method according to any one of claims 1 to 4, characterized in that: The receiving a first message set sent by the second device includes: In a case where the first message set meets a sending condition, the first message set sent by the second device is received, where the sending condition is associated with a message size and / or message accumulation time of the first message set.

6. A message processing method, characterized in that: Applied to the second device, comprising: Generate a first message set, the first message set including at least two messages, the first message set being used to generate a first index associated with the first message set, the first index being used to pull messages from a data log; The first message set is sent to a first device.

7. The method according to claim 6, characterized in that The first index includes one or more of the following: first information, the first information being used to indicate a position of the first message set in the data log; second information, wherein the second information represents the number of bytes of the first message set; third information, where the third information is used to indicate a label of the first message set; fourth information, where the fourth information is used to indicate a position of the first index in the index file; The fifth information is used to indicate the number of messages in the first message set.

8. The method according to claim 6, characterized in that The sending the first message set to the first device includes: In a case where the first message set meets a sending condition, the first message set is sent to the first device, where the sending condition is associated with a message size and / or message accumulation time of the first message set.

9. The method according to claim 8, characterized in that The generating of the first message set comprises: Writing the at least two messages in sequence through multiple threads to obtain the first message set, each thread being used to write one message; Determine whether the first message set meets the sending condition.

10. The method according to claim 9, characterized in that The multiple threads include a first thread, the first thread is at least used to write a first message, and the at least two messages are written in sequence through the multiple threads to obtain the first message set, including: Setting lock information for the first message set by the first thread, wherein the lock information is used to lock the first message set; The first message is written by the first thread.

11. A first device, characterized in that: include: a first processor and a first memory for storing a computer program executable on the processor, Wherein, when the first processor is used to run the computer program, the steps of the method described in any one of claims 1 to 5 are executed.

12. A second device, characterized in that: include: a second processor and a second memory for storing a computer program executable on the processor, Wherein, when the second processor is used to run the computer program, the steps of the method described in any one of claims 6 to 10 are executed.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 10.

Images