Message processing method, apparatus, device, storage medium and computer program product

Abstract

This application discloses a message processing method, apparatus, device, storage medium, and computer program product, wherein the method is applied in a message consumer service, and the message consumer service and message producer service are associated through a distributed storage cluster. The method includes: determining a target business object to be processed from an ordered list; determining whether the target business object has reached a set processing time based on the most recent timestamp information of the target business object; when the target business object reaches the set processing time, setting a distributed lock on the target business object; obtaining a target ordered message queue corresponding to the target business object from the distributed storage cluster; determining a target thread that has obtained the distributed lock, and using the target thread to process the operation messages in the target ordered message queue in sequence. Single-point serialization storage can ensure the sequential processing of messages and avoid the occurrence of timestamp out-of-order situations.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a message processing method, a message processing apparatus, a business processing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of mobile communication technology, the need to maintain the orderly processing of various user messages in the context of microservice architecture is increasing. Ordered messaging means that messages must be consumed in a specified order during processing. For example, in e-commerce scenarios, the processing of various order statuses for user purchases needs to be ordered: Order generated > Order pending payment > Order payment successful > Order pending shipment > Order pending receipt > Order completed. If there are after-sales issues, there might also be order refund processing > Order refund completed, etc. These order statuses must be processed strictly in order. However, in high-availability microservice scenarios (multiple service nodes) with concurrency, order operations may be handled by different service nodes at different statuses, causing system anomalies and chaotic status data processing, leading to disordered timing. Summary of the Invention

[0003] This application provides a message processing method, apparatus, device, storage medium, and computer program product to ensure the sequential processing of messages and avoid data corruption.

[0004] According to one aspect of this application, a message processing method is provided, which is applied in a message consumer service. The message consumer service and the message producer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object. The ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue is used to record the operation messages to be processed in the corresponding business objects and the unique message identifier of the operation messages.

[0005] The method includes:

[0006] The target business object to be processed is determined from the ordered list;

[0007] Based on the most recent timestamp information of the target business object, determine whether the target business object has reached the set processing time.

[0008] When the target business object reaches the set processing time, a distributed lock is set for the target business object;

[0009] Obtain the target ordered message queue corresponding to the target business object from the distributed storage cluster;

[0010] The target thread that acquires the distributed lock is determined, and the target thread processes the operation messages in the target ordered message queue in sequence.

[0011] According to another aspect of this application, a message processing method is provided, which is applied in a message producer service. The message producer service and the message consumer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object. The ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue is used to record the operation messages to be processed in the corresponding business objects and the unique message identifier of the operation messages.

[0012] The method includes:

[0013] In response to an operation initiated by a business instance on a business object, an operation message is generated, and a unique message identifier is generated for the operation message based on a single point-of-sale (SOS) mechanism.

[0014] Obtain the operation timestamp information that initiated the operation, and record the identifier of the business object and the operation timestamp information in an ordered list in the distributed storage cluster;

[0015] Obtain the ordered message queue corresponding to the business object in the distributed storage cluster, and write the operation message and the unique message identifier into the ordered message queue.

[0016] According to another aspect of this application, a message processing apparatus is provided, which is applied in a message consumer service. The message consumer service and the message producer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object. The ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue is used to record the operation messages to be processed in the corresponding business objects and the unique message identifier of the operation messages.

[0017] The device includes:

[0018] The target business object determination module is used to determine the target business object to be processed from the ordered list;

[0019] The processing timing determination module is used to determine whether the target business object has reached the set processing timing based on the most recent timestamp information of the target business object;

[0020] The distributed lock addition module is used to set a distributed lock on the target business object when the target business object reaches the set processing time.

[0021] The target ordered message queue acquisition module is used to acquire the target ordered message queue corresponding to the target business object in the distributed storage cluster.

[0022] The message processing module is used to determine the target thread that acquires the distributed lock, and to process the operation messages in the target ordered message queue in sequence using the target thread.

[0023] According to another aspect of this application, a message processing apparatus is provided, applied in a message producer service, wherein the message producer service and the message consumer service are associated through a distributed storage cluster, wherein the distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object, wherein the ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information, and the ordered message queue is used to record the operation messages to be processed in the corresponding business objects and the unique message identifier of the operation messages.

[0024] The device includes:

[0025] The operation message generation module is used to generate operation messages in response to operations initiated by business instances on business objects.

[0026] The message identifier generation module is used to generate a unique message identifier for the operation message based on a single point-of-sale (SOS) mechanism.

[0027] The business object writing module is used to obtain the operation timestamp information of the operation that initiated the operation, and record the identifier of the business object and the operation timestamp information in an ordered list of the distributed storage cluster.

[0028] The operation message writing module is used to obtain the ordered message queue corresponding to the business object in the distributed storage cluster, and write the operation message and the unique message identifier into the ordered message queue.

[0029] According to another aspect of this application, a service processing device is provided, the service processing device comprising:

[0030] At least one processor; and

[0031] A memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform a message processing method according to any embodiment of this application.

[0033] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement a message processing method as described in any embodiment of this application.

[0034] According to another aspect of this application, a computer program product is provided, the computer program product including computer-executable instructions, which, when executed, are used to implement a message processing method as described in any embodiment of this application.

[0035] In this embodiment, when the message producer service generates an operation message in response to an operation initiated by a business instance on a business object, a single-point sequence number generator is used to generate a unique message identifier for the operation message. This avoids the situation where the generated timestamps are out of order due to inconsistencies in the clocks of multiple clients and multiple servers. Then, when writing the operation message, message recording is performed using two core storage structures: an ordered list and an ordered message queue stored in the distributed storage cluster. The identifier of the business object and the operation timestamp information of the current operation are recorded in the ordered list, and the operation message and its unique message identifier are recorded in the ordered message queue. This achieves single-point serialization storage and avoids the impact of inconsistencies in the clocks of multiple clients and multiple servers.

[0036] Distributed storage clusters can decouple message producer and message consumer services, ensuring that when a message fails to send and is retried, other message production during the retry period does not affect message order processing; or, when network transmission is unstable, messages sent first (or in priority but arriving later) do not affect message order processing. Furthermore, it facilitates the upgrading or expansion of both message producer and message consumer services without affecting message order processing during upgrades or expansions. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a message processing method provided in Embodiment 1 of this application;

[0039] Figure 2 This is a message processing system architecture diagram provided in Embodiment 1 of this application;

[0040] Figure 3 Embodiment 2 of this application provides a flowchart of a message processing method;

[0041] Figure 4 This is a schematic diagram of an orderly processing method for paper message provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of a message processing device provided in Embodiment 3 of this application;

[0043] Figure 6 This is a schematic diagram of the structure of a message processing device provided in Embodiment 4 of this application;

[0044] Figure 7 This is a schematic diagram of the structure of a service processing device that implements a message processing method according to an embodiment of this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0046] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] Example 1

[0048] Figure 1 The flowchart provided in Embodiment 1 of this application illustrates a message processing method that can be applied to message producer services. For example... Figure 2 As shown in the message processing system architecture diagram, the message producer service and message consumer service are associated through a distributed storage cluster.

[0049] The message producer service serves as the system's front-end entry point, providing various user-operable functions to front-end users. Multiple message producer services can exist, forming a message producer service cluster, which can rapidly scale to increase message production speed.

[0050] In one embodiment, the distributed storage cluster may include a Codis cluster, which is a middleware for storing messages, based on memory, and offering fast read and write speeds. Codis is a distributed Redis (high-performance key-value database) solution. For upper-layer applications, connecting to the Codis Proxy (which provides the entry point for connecting to the clustered Redis service and acts as a proxy service for client connections; the Codis Proxy is a single node) is not significantly different from connecting to the native Redis Server. Upper-layer applications can use it as if it were a single-machine Redis instance. Codis handles request forwarding, non-disruptive data migration, and other tasks at the underlying level. All of this is transparent to the client; it can be simply understood that Codis is connected to a Redis service with virtually unlimited memory.

[0051] Message consumer services are used to retrieve data from a distributed storage cluster and process it according to business logic. Multiple message consumer services can exist, forming a message consumer service cluster, which can be rapidly scaled up to increase message consumption speed.

[0052] like Figure 1 As shown, this embodiment may include the following steps:

[0053] Step 110: In response to the operation initiated by the business instance on the business object, an operation message is generated, and a unique message identifier is generated for the operation message based on the single sign-on mechanism.

[0054] Specifically, when a business instance initiates an operation on a business object and calls the message producer service, the message producer service can generate a corresponding operation message for the operation and generate a unique message identifier for the operation message.

[0055] In this embodiment, the business instance, business object, and operations can vary depending on the specific business scenario, and this embodiment does not impose any limitations on these aspects. For example, in an e-commerce scenario, the business object can be a product, and the operations can include placing an order, making a payment, or shipping the product. Similarly, in the short video or live streaming field, the business object can be a specific video object within the short video or live streaming domain, and the operations can include clicking, publishing, or following. Alternatively, the business object can also be a gameplay object, and the operations can include creating or viewing the gameplay object.

[0056] In one embodiment, if the distributed storage cluster is a Codis cluster, then step 110, which generates a unique message identifier for the operation message, may further include the following steps:

[0057] The INCR instruction based on the Codis cluster is used as a sequence number generator to generate a unique message identifier for the operation message.

[0058] In this embodiment, to avoid out-of-order message generation, Codis-union is used as the clock benchmark, and the INCR instruction based on the Codis cluster is used as the sequence number generator. The unique message identifier of the current operation message is obtained by auto-incrementing using the identifier of the business object as the key. The INCR instruction is used to perform an atomic increment operation on the value stored in the specified key. If the specified key does not exist, its value will be set to 0 before executing the INCR operation. If the value stored in the specified key is not a string type or the stored string type cannot be represented as an integer, the server will return an error code when executing this command. By using Codis as the sequence number generator, restarting the message consumer service will not affect sequential consumption.

[0059] In other embodiments, a single-point numbering mechanism can be implemented using a MySQL database for storing the numbering information or a ZooKeeper registry center for issuing numbers. However, compared to using a Codis union cluster based on memory and hashing the business object's identifier to a specific Redis node using the identifier as the key, the Codis union cluster outperforms other alternatives in performance and can support a higher QPS (Query Per Second) limit.

[0060] In a further embodiment, when generating a unique message identifier for the current operation message, the target sorting dimension can be determined first, and a unique message identifier related to the target sorting dimension can be generated for the operation message, thereby supporting sorting requirements of different dimensions.

[0061] The target sorting dimension can be a preset sorting dimension related to the business scenario. Different business scenarios can set different sorting dimensions. For example, the target sorting dimension can include time order dimension, state order dimension, etc.

[0062] Once the target sorting dimension is determined, the unique message identifier of the current operation message is associated with its target sorting dimension. For example, if the target sorting dimension is a time order dimension, to avoid clock inconsistencies among multiple clients and servers, which could lead to out-of-order timestamps generated by messages processed by different service nodes, the Codis cluster's INCR auto-incrementing sequence number generator is used. Each message producer service node receives a client request and increments the sequence number of the current operation message using the identifier of the current business object as the key. The order of this sequence number is absolutely ordered with the message production time, eliminating the impact of clock inconsistencies. In implementation, the Seq sequence number can be used as the sequence number returned by the sequence number generator. As another example, if the target sorting dimension is a state order dimension, such as the need for ordered processing of various states of a user's order for purchasing goods, strictly following the order of Order Generation > Order Pending Payment > Order Payment Successful > Order Pending Shipment > Order Pending Receipt > Order Completed, and if there are after-sales situations, it may also include processes such as Order Refund Processing > Order Refund Completed. In this scenario, an ordered sequence number mapping for the state can be set, and the Seq sequence number becomes the sequence number of that state.

[0063] In practical applications, users can change the target sorting dimension midway through message production. Subsequent Seq numbers can then be generated based on the changed target sorting dimension. Seq numbers generated according to the changed target sorting dimension not only meet business requirements but also maintain compatibility with existing data. For example, if the Seq number generated according to the time-order dimension is timestamp 1648189672, and the business wants to change it to the status-order dimension after running it using the time-order dimension for a period, the Seq number generation rule can be modified, for example, to (status + timestamp). That is, the first few bits of the timestamp 1648189672 are defined as the status value, and the number of bits is padded accordingly. For example, if the status is 11, the Seq number after adding the status to the timestamp would be 111648189672. The Seq number generated according to the new sorting dimension can then be directly recorded in the original ordered message queue.

[0064] Step 120: Obtain the operation timestamp information that initiated the operation, and record the identifier of the business object and the operation timestamp information in an ordered list of the distributed storage cluster.

[0065] One of the core storage structures included in the Codis cluster in this embodiment is an ordered list, used to record multiple pending business objects and their most recent operation timestamp information. Specifically, the ordered list stores a list of changed business objects, where the member is the identifier of the business object, and the score is the operation timestamp information that initiated the current operation (used to indicate the last update timestamp of the current business object). Therefore, for each operation, the message producer service can obtain the operation timestamp information that initiated the current operation, and then write this operation timestamp information and the identifier of the current business object into the ordered list. Since the ordered queue is stored in the Codis cluster, the impact of inconsistent clocks among service nodes can be avoided when reading data subsequently.

[0066] In one implementation, the timestamp information of the current operation can be obtained by using the TIME function.

[0067] Step 130: Obtain the ordered message queue corresponding to the business object in the distributed storage cluster, and write the operation message and the unique message identifier into the ordered message queue.

[0068] Another core storage structure included in the Codis cluster in this embodiment is an ordered message queue. Each business object has a corresponding ordered message queue, which records the operation messages to be processed in its corresponding business object and the unique message identifier of each operation message. Specifically, the ordered message queue is used to store the specific message content and its sorting dimension of each business object. This structure ensures that messages of the same business object are processed serially. In the ordered message queue, its member is the message body of the current operation message, and its score is the unique message identifier (i.e., the Seq number) of the current operation message.

[0069] In this embodiment, when the message producer service generates an operation message in response to an operation initiated by a business instance on a business object, a single-point sequence number generator is used to generate a unique message identifier for the operation message. This avoids the situation where the generated timestamps are out of order due to inconsistencies in the clocks of multiple clients and multiple servers. Then, when writing the operation message, message recording is performed using two core storage structures: an ordered list and an ordered message queue stored in the distributed storage cluster. The identifier of the business object and the operation timestamp information of the current operation are recorded in the ordered list, and the operation message and its unique message identifier are recorded in the ordered message queue. This achieves single-point serialization storage and avoids the impact of inconsistencies in the clocks of multiple clients and multiple servers.

[0070] Distributed storage clusters can decouple message producer and message consumer services, ensuring that when a message fails to send and is retried, other message production during the retry period does not affect message order processing; or, when network transmission is unstable, messages sent first (or in priority but arriving later) do not affect message order processing. Furthermore, it facilitates the upgrading or expansion of both message producer and message consumer services without affecting message order processing during upgrades or expansions.

[0071] Example 2

[0072] Figure 3 This application provides a flowchart of a message processing method according to Embodiment 2. This embodiment can be applied to a message consumer service, which is associated with a message producer service through a distributed storage cluster. The distributed storage cluster stores an ordered list and ordered message queues corresponding to each business object. The ordered list records multiple pending business objects and their most recent operation timestamps. The ordered message queues record pending operation messages within their corresponding business objects and the unique message identifiers of each operation message. Figure 3 As shown, this embodiment may include the following steps:

[0073] Step 210: Determine the target business object to be processed from the ordered list.

[0074] In this step, if there are multiple unprocessed business objects in the ordered list, the target business object is selected for processing according to the set strategy.

[0075] In one embodiment, step 210 may further include the following steps:

[0076] Select the business object with the earliest recent operation timestamp as the target business object to be processed.

[0077] Among them, the earliest recent operation timestamp refers to the timestamp with the longest time difference from the current time. In implementation, the recent operation timestamps of each record in the ordered list can be compared to find the earliest one. Then, the business object with the earliest recent operation timestamp is selected as the target business object to be processed. At this point, the identifier and recent operation timestamp information of the target business object can be obtained.

[0078] Step 220: Based on the most recent timestamp information of the target business object, determine whether the target business object has reached the set processing time for performing order verification.

[0079] In this step, the set processing timing is related to the time the system can tolerate for message retransmission failures. Within the processing timing, message retransmission is allowed to ensure that retransmitted messages reach the ordered message queue; once the processing time arrives, message retransmission is no longer allowed, and message processing for the business object begins. In one embodiment, step 220 may further include the following steps:

[0080] Calculate the time difference between the current time and the most recent operation timestamp of the target business object; if the time difference is greater than or equal to a preset delay processing time threshold, determine that the target business object has reached the set processing time; if the time difference is less than the preset delay processing time threshold, determine that the target business object has not reached the set processing time.

[0081] It should be noted that the delay processing time threshold is the maximum time the system can tolerate for message failure retries. Different delay processing time thresholds can be set according to different business requirements; this embodiment does not impose any limitations on this.

[0082] If the time difference between the current time and the timestamp of the most recent operation of the target business object exceeds a preset delay processing time threshold, it means that the target business object is no longer allowed to retry upon failure. At this time, the target business object reaches the processing time, and step 230 can be further executed. If the time difference between the current time and the timestamp of the most recent operation of the target business object does not exceed the preset delay processing time threshold, it means that the target business object is allowed to continue to retry upon failure within this threshold time. At this time, no processing is required on the target business object until the preset delay processing time threshold is reached. This ensures that the message content of the same business object has arrived in the ordered message queue when the network is unstable or the message fails and is retransmitted, avoiding the problem of out-of-order messages not being consumed immediately due to network instability or failure retry, and ensuring the orderly consumption of messages.

[0083] In practice, if strict consistency is required for orderliness, an alert can be issued after N failed retries to allow manual intervention on the target business object, or a pre-processing state check can be performed during message consumption.

[0084] Step 230: When the target business object reaches the set processing time, a distributed lock is set for the target business object.

[0085] When it is necessary to consume the current target business object, a distributed lock is set for the target business object to ensure that only one thread processes the messages of the same business object serially, thereby ensuring the orderliness of message processing.

[0086] Additionally, when it is necessary to consume a target business object, the record corresponding to that target business object can be deleted from the ordered list.

[0087] Step 240: Obtain the target ordered message queue corresponding to the target business object in the distributed storage cluster.

[0088] When consuming a target business object begins, the corresponding ordered message queue for that object can be located in the Codis cluster. This ordered message queue stores multiple pending operation messages for the target business object, along with a unique message identifier for each message. This unique message identifier can be a sequence number generated according to the Codis cluster's single-point numbering mechanism and a set sorting dimension.

[0089] Step 250: Determine the target thread that acquires the distributed lock, and use the target thread to process the operation messages in the target ordered message queue in sequence.

[0090] In one implementation, the target thread can be a thread used to process the current target business object. Based on the business requests of each thread, the privileges of the distributed lock can be assigned to one of these threads; this thread becomes the target thread. The target thread can then process the operation messages of the target business object sequentially. For example, the target thread can retrieve messages from an ordered message queue and process them in ascending order of their Seq numbers.

[0091] Once all operation messages for the target business object have been processed, the distributed lock of the target business object can be deleted.

[0092] In this embodiment, during message consumption, the target business object is first determined from the ordered list. Then, the order of the target business object is verified. If the order verification passes, a distributed lock is set for the target business object to ensure that only one thread processes messages from the same business object serially, thus guaranteeing the orderliness of message processing. Upon starting message consumption, the target ordered message queue of the target business object is obtained, and messages are processed sequentially according to the unique message identifier of each operation message in the target ordered message queue, further ensuring ordered message processing.

[0093] In addition, Codis clusters allow message consumption services to scale up infinitely, without being limited by the number of partitions.

[0094] To enable those skilled in the art to better understand this application, the following is an exemplary illustration through a specific application scenario:

[0095] This application scenario example uses a live streaming platform as an example. Assume a live streaming platform offers a "paper slip" game, which includes sending paper slips, drawing paper slips, and leaving comments on the drawn slips. When a user wants to send a paper slip, they can click the "Send Paper Slip" button on the homepage and fill in the relevant information on the pop-up paper slip sending page. When a user wants to draw a paper slip, they can click the "Draw Paper Slip" button on the homepage. If a paper slip is successfully drawn, the system will notify the sender that the paper slip has been drawn and update the number of drawn slips. When a user leaves a comment on a drawn paper slip, the system will notify the sender that the paper slip has been commented on and update the number of comments on the paper slip. A single paper slip can be drawn or commented on by multiple people, and each paper slip will have a corresponding number of drawn slips and comments.

[0096] The product requirement necessitates the sequential processing of message retrieval and message delivery for a single note. To achieve this, three modules were designed: a note service, a Codis cluster, and a message processing service.

[0097] The note service acts as the message producer, serving front-end users by providing functions such as sending, retrieving, and leaving messages. It serves as the system's front-end entry point. When users retrieve notes or leave messages, the note service processes these events, generating corresponding messages and writing them to the Codis cluster. This module can quickly scale to improve message production speed. In this example, the note is equivalent to the business object in this application; sending, retrieving, and leaving messages on notes are equivalent to operations initiated on the business object.

[0098] Codis Cluster: A middleware for storing messages, based on memory, offering fast read and write speeds. It includes two core storage structures: Note-Change-List (a sorted set structure) and Note-Message-Sequence (also a sorted set structure). Note-Change-List stores a list of changed notes, where member is the note ID and score is the last update timestamp for that note ID. Note-Message-Sequence stores the specific message content of each note and its sorting dimensions (time, status, or other scenario dimensions). This structure ensures that messages for the same note are processed serially.

[0099] Message processing service: This is the role of the message consumer, equivalent to a message consumer service. It retrieves data from the Codis cluster, extracts changed messages, performs ordered verification on those messages, and processes the messages sequentially to execute business logic. This module can quickly scale to improve message consumption speed.

[0100] refer to Figure 4 This diagram illustrates the orderly processing of note messages, including the message production and consumption processes as follows:

[0101] The message production process for the note service is as follows:

[0102] 1. When a user performs operations such as drawing or leaving a message on a note, the note service in the note service cluster (including NoteService 1, Note Service 2, ..., Note Service N) is invoked;

[0103] 2. The note service generates note messages and uses a sequence number generator (seq-generate) based on a Codis-union cluster to generate the sequence number (seq) of the note message. The generated seq number is related to the sorting dimension, which includes the time order dimension and other order dimensions.

[0104] 1) Time sequence dimension

[0105] To avoid out-of-order timestamps generated by messages processed by different service nodes due to clock inconsistencies among multiple clients and servers, the Codis cluster's INCR auto-incrementing sequence number generator is used. Each message service node receives a client request and increments the sequence number (seq) of the current message using the message ID as the key. The order of these sequence numbers is absolutely ordered according to the message production time, eliminating the influence of clock inconsistencies. The seq number is the sequence number returned by the sequence number generator.

[0106] 2) Other order dimensions

[0107] For example, in terms of the status dimension, the order status needs to strictly follow the order of order pending payment > order payment successful > order pending shipment > order pending receipt > order, etc. Set an ordered sequence number for the status mapping, and the seq number is the sequence number of the status.

[0108] 3. Write the change record on the paper note

[0109] Write the change information of the note operation into the Codis Note-Change-List (Sorted set), where member is the note ID (e.g., ...). Figure 4 In the list of notes (note1-note7), the score is the last update timestamp of that note ID (i.e., the most recent operation timestamp). This avoids the impact of inconsistent clocks across service nodes. In implementation, the current timestamp information is obtained using TIME through Codis.

[0110] 4. Write the corresponding note's message into the ordered queue of note messages.

[0111] Each note maintains an ordered message queue, Note-Message-Sequence, and messages are written to Note-Message-Sequence (an ordered set), where members are the message bodies (e.g., ...). Figure 4 In the sequence (msg1-msg2), score is the seq number generated in step 2 (e.g., msg1-msg2). Figure 4 In this context, seq represents the order of the sorting dimensions.

[0112] The message consumption process of the message processing service is as follows:

[0113] 5. The message processing service retrieves the note change information from the Note-Change-List in the Codis cluster;

[0114] 6. The thread that retrieves the paper strip processes the message content in Note-Message-Sequence in an orderly manner.

[0115] The specific processes of the two consumption processes mentioned above are as follows:

[0116] 1) The message processing service retrieves the note information with the shortest time from the Note-Change-List in the Codis cluster.

[0117] 2) Extract the note ID (noteId) and the last update timestamp (time) of the note from the note with the shortest time.

[0118] 3) Check if the current time minus `time` is greater than the delayed processing time threshold (configurable for each scenario) to ensure that messages for the same note have arrived in the ordered set of Note-Message-Sequence even when the network is unstable or messages fail and are resent, thus avoiding the impact of out-of-order delivery caused by network instability and message failure resentment. If the current time minus `time` is greater than the delayed processing time threshold, continue processing; otherwise, do not process and repeat step 1).

[0119] 4) Add a distributed lock to the message processing for the note ID.

[0120] 5) Remove the note ID from the Note-Change-List to ensure that messages with that note ID are only processed serially by the same thread, thus guaranteeing the orderliness of message processing.

[0121] 6) The thread that acquires the distributed lock retrieves the Note-Message-Sequence for the note ID and then processes the message according to the sequence number.

[0122] 7) Delete the distributed lock for message processing of the note ID.

[0123] This example uses a Codis-union cluster as a single-point sequencer for each message, a sorted-set structure to ensure single-point serialization, and delayed processing threshold verification and distributed locks to ensure that only one thread processes the message of the same message, thus achieving high-throughput processing with guaranteed message order.

[0124] Example 3

[0125] Figure 5 This is a schematic diagram of a message processing device provided in Embodiment 3 of this application, which is applied in a message consumer service. The message consumer service and the message producer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object. The ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue is used to record the operation messages to be processed in the corresponding business objects and the unique message identifier of the operation messages.

[0126] like Figure 5 As shown, this embodiment may include the following modules:

[0127] The target business object determination module 310 is used to determine the target business object to be processed from the ordered list;

[0128] The processing timing determination module 320 is used to determine whether the target business object has reached the set processing timing based on the most recent timestamp information of the target business object;

[0129] The distributed lock adding module 330 is used to set a distributed lock on the target business object when the target business object reaches the set processing time.

[0130] The target ordered message queue acquisition module 340 is used to acquire the target ordered message queue corresponding to the target business object in the distributed storage cluster.

[0131] The message processing module 350 is used to determine the target thread that acquires the distributed lock, and to use the target thread to process the operation messages in the target ordered message queue in sequence.

[0132] In one embodiment, the target business object determination module 310 is specifically used for:

[0133] Select the business object with the earliest recent operation timestamp as the target business object to be processed.

[0134] In one embodiment, the processing timing determination module 320 is specifically used for:

[0135] Calculate the time difference between the current time and the most recent operation timestamp of the target business object;

[0136] If the time difference is greater than or equal to the preset delay processing time threshold, then the target business object is determined to have reached the set processing time.

[0137] If the time difference is less than the preset delay processing time threshold, it is determined that the target business object has not reached the set processing time.

[0138] In one embodiment, the device may further include the following modules:

[0139] The distributed lock deletion module is used to delete the distributed lock after all operation messages of the target business object have been processed.

[0140] In one embodiment, the message processing module 350 is specifically used for:

[0141] The target thread retrieves messages from the ordered message queue in ascending order of the unique message identifier and processes them.

[0142] In one embodiment, the distributed storage cluster includes a Codis cluster, and the unique message identifier is a sequence number generated according to the single-point numbering mechanism of the Codis cluster and a set sorting dimension.

[0143] In one embodiment, the device may further include the following modules:

[0144] The business object deletion module is used to delete the record corresponding to the target business object from the ordered list after the distributed lock is set on the target business object.

[0145] The message processing apparatus provided in this application embodiment can execute a message processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0146] Example 4

[0147] Figure 6 This is a schematic diagram of a message processing device provided in Embodiment 4 of this application, which is applied in a message producer service. The message producer service and the message consumer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object. The ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue is used to record the operation messages to be processed in the corresponding business objects and the unique message identifier of the operation messages.

[0148] like Figure 6 As shown, this embodiment may include the following modules:

[0149] The operation message generation module 410 is used to generate operation messages in response to operations initiated by business instances on business objects.

[0150] The message identifier generation module 420 is used to generate a unique message identifier for the operation message based on a single-point numbering mechanism;

[0151] The business object writing module 430 is used to obtain the operation timestamp information of the operation that initiated the operation, and record the identifier of the business object and the operation timestamp information in an ordered list of the distributed storage cluster.

[0152] The operation message writing module 440 is used to obtain the ordered message queue corresponding to the business object in the distributed storage cluster, and write the operation message and the unique message identifier into the ordered message queue.

[0153] In one embodiment, the distributed storage cluster includes a Codis cluster, and the message identifier generation module 420 is specifically used for:

[0154] The INCR instruction based on the Codis cluster is used as a sequence number generator to generate a unique message identifier for the operation message.

[0155] In a further embodiment, the message identifier generation module 420 is specifically used for:

[0156] Determine the target ranking dimensions;

[0157] Generate a unique message identifier for the operation message that is related to the target sorting dimension.

[0158] The message processing apparatus provided in this application embodiment can execute a message processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0159] Example 5

[0160] Figure 7 A schematic diagram of the structure of a service processing device 10 that can be used to implement an embodiment of the method of this application is shown. Figure 7As shown, the service processing device 10 includes at least one processor 11 and a storage device, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The storage device stores one or more computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the service processing device 10.

[0161] Multiple components in the service processing device 10 are connected to the I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of display, speaker, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless communication transceiver, etc.

[0162] The processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities, used to execute the various methods and processes described above, such as a message processing method.

[0163] In some embodiments, a message processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the business processing device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the message processing method described above may be performed.

[0164] In some embodiments, a message processing method may be implemented as a computer program product including computer-executable instructions that, when executed, perform one or more steps of the message processing method described above.

Claims

1. A message processing method, characterized in that, In a message consumer service, the message consumer service and the message producer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object. The ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue is used to record the operation messages to be processed in the corresponding business object and the unique message identifier of the operation message. Each business object has a corresponding ordered message queue. The method includes: The target business object to be processed is determined from the ordered list; Calculate the time difference between the current time and the most recent operation timestamp of the target business object; If the time difference is greater than or equal to the preset delay processing time threshold, then the target business object is determined to have reached the set processing time. If the time difference is less than the preset delay processing time threshold, it is determined that the target business object has not reached the set processing time. When the target business object reaches the set processing time, a distributed lock is set for the target business object; Obtain the target ordered message queue corresponding to the target business object from the distributed storage cluster; The target thread that acquires the distributed lock is determined, and the target thread processes the operation messages in the target ordered message queue in sequence.

2. The method according to claim 1, characterized in that, The step of determining the target business object to be processed from the ordered list includes: Select the business object with the earliest recent operation timestamp as the target business object to be processed.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Once all operation messages for the target business object have been processed, the distributed lock is deleted.

4. The method according to claim 1, characterized in that, The step of processing messages in the ordered message queue sequentially using the target thread includes: The target thread retrieves messages from the ordered message queue in ascending order of the unique message identifier and processes them.

5. The method according to claim 1, characterized in that, After setting the distributed lock on the target business object, the method further includes: Delete the record corresponding to the target business object from the ordered list.

6. A message processing method, characterized in that, This is applied to a message producer service, where the message producer service and message consumer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and ordered message queues corresponding to each business object. The ordered list records multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue records the operation messages to be processed in the corresponding business object and the unique message identifier of the operation message. Each business object has a corresponding ordered message queue. The method includes: In response to an operation initiated by a business instance on a business object, an operation message is generated, and a unique message identifier is generated for the operation message based on a single point-of-sale (SOS) mechanism. Obtain the operation timestamp information that initiated the operation, and record the identifier of the business object and the operation timestamp information in an ordered list in the distributed storage cluster; Obtain the ordered message queue corresponding to the business object in the distributed storage cluster, and write the operation message and the unique message identifier into the ordered message queue.

7. The method according to claim 6, characterized in that, The process of generating a unique message identifier for the operation message based on a single point-of-sale (SOS) mechanism includes: Determine the target ranking dimensions; The INCR instruction based on a Codis cluster is used as a sequence number generator to generate a unique message identifier for the operation message that is related to the target sorting dimension.

8. A message processing device, characterized in that, In a message consumer service, the message consumer service and the message producer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and an ordered message queue corresponding to each business object. The ordered list is used to record multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue is used to record the operation messages to be processed in the corresponding business object and the unique message identifier of the operation message. Each business object has a corresponding ordered message queue. The device includes: The target business object determination module is used to determine the target business object to be processed from the ordered list; The processing timing determination module is used to calculate the time difference between the current time and the most recent operation timestamp information of the target business object; If the time difference is greater than or equal to the preset delay processing time threshold, then the target business object is determined to have reached the set processing time. If the time difference is less than the preset delay processing time threshold, it is determined that the target business object has not reached the set processing time. The distributed lock addition module is used to set a distributed lock on the target business object when the target business object reaches the set processing time. The target ordered message queue acquisition module is used to acquire the target ordered message queue corresponding to the target business object in the distributed storage cluster. The message processing module is used to determine the target thread that acquires the distributed lock, and to process the operation messages in the target ordered message queue in sequence using the target thread.

9. A message processing device, characterized in that, This is applied to a message producer service, where the message producer service and message consumer service are associated through a distributed storage cluster. The distributed storage cluster stores an ordered list and ordered message queues corresponding to each business object. The ordered list records multiple business objects to be processed and their most recent operation timestamp information. The ordered message queue records the operation messages to be processed in the corresponding business object and the unique message identifier of the operation message. Each business object has a corresponding ordered message queue. The device includes: The operation message generation module is used to generate operation messages in response to operations initiated by business instances on business objects. The message identifier generation module is used to generate a unique message identifier for the operation message based on a single point-of-sale (SOS) mechanism. The business object writing module is used to obtain the operation timestamp information of the operation that initiated the operation, and record the identifier of the business object and the operation timestamp information in an ordered list of the distributed storage cluster. The operation message writing module is used to obtain the ordered message queue corresponding to the business object in the distributed storage cluster, and write the operation message and the unique message identifier into the ordered message queue.

10. A business processing device, characterized in that, The business processing equipment includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

12. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1-7.

Images