Message processing method, system, equipment, medium and product

Abstract

The invention provides a message processing method, system and device, a medium and a product, which can be applied to the distributed technical field and the financial science and technology field. The method is applied to a preset distributed system. The preset distributed system comprises a message source end, a message target end and a consumption request processing end; the method comprises the following steps that: the message source end synchronizes messages in a first message queue to a second message queue of the message target end; aiming at a consumption request of a target consumer, the consumption request processing end selects to process the aiming consumption request based on the first message queue or the second message queue; and under the condition of determining that the consumption request is processed based on the second message queue, determining a target message corresponding to the current consumption progress of the target consumer in the second message queue, and processing the consumption request based on a message after the target message in the second message queue.

Description

Technical Field

[0001] This application relates to the fields of distributed technology and financial technology, specifically to a message processing method, system, device, medium, and product. Background Technology

[0002] Currently, message subscription mechanisms are widely used in business applications. In this mechanism, producers and consumers communicate asynchronously through message service nodes. Producers publish messages to the message service nodes, and consumers retrieve the messages published by the producers from the message service nodes. Message service nodes can store and distribute messages through message topics or message queues.

[0003] However, the current stability of individual message service nodes is poor, resulting in poor stability for consumers when consuming messages. Summary of the Invention

[0004] In view of the above problems, this application provides a message processing method, system, device, medium and product to improve the stability of consumers when consuming messages.

[0005] According to a first aspect of this application, a message processing method is provided, applied to a pre-defined distributed system; the pre-defined distributed system includes: a message source end, a message destination end, and a consumption request processing end; the method includes: the message source end synchronizing messages in a first message queue to a second message queue of the message destination end; the consumption request processing end, for a consumption request of a target consumer, selecting to process the target consumption request based on either the first message queue or the second message queue; if it is determined that the target consumption request is processed based on the second message queue, determining the target message in the second message queue corresponding to the current consumption progress of the target consumer, and processing the target consumption request based on messages following the target message in the second message queue.

[0006] Optionally, the current consumption progress of the target consumer includes: the latest message consumed by the target consumer in the first message queue; determining the target message corresponding to the current consumption progress of the target consumer in the second message queue includes: determining the message corresponding to the latest message consumed by the target consumer in the first message queue in the second message queue as the target message based on the offset mapping relationship of the same message between the first message queue and the second message queue.

[0007] Optionally, the current consumption progress of the target consumer includes: the identification information of the latest consumed message of the target consumer; determining the target message corresponding to the current consumption progress of the target consumer in the second message queue includes: determining the message corresponding to the identification information of the latest consumed message of the target consumer in the second message queue as the target message.

[0008] Optionally, when determining the consumption request to be processed based on the second message queue, the method further includes: the consumption request processing terminal determining in real time the offset mapping relationship of the same message between the first message queue and the second message queue.

[0009] Optionally, the message source end synchronizes messages in the first message queue to the second message queue of the message target end, including: the message source end performing message synchronization operation and message verification operation in parallel; the message synchronization operation includes: synchronizing messages in the first message queue of the message source end to a preset storage area in the message target end; the message verification operation includes: under a preset condition, generating integrity verification information for a sequence of messages to be verified in the first message queue that is not currently synchronized to the second message queue but is currently synchronized to the preset storage area, and the message order information in the sequence of messages to be verified, and sending the generated integrity verification information to the message target end; if the message target end determines, based on the received integrity verification information, that the sequence of messages to be verified in the preset storage area and the message order information have passed integrity verification, storing the sequence of messages to be verified in the preset storage area into the second message queue according to the message order information.

[0010] Optionally, the selection of the consumption request based on the first message queue or the second message queue includes any one of the following: determining the consumption request based on the second message queue when the number of consumers for the first message queue is greater than a preset consumer number threshold; determining the consumption request based on the second message queue when the number of messages between the current consumption progress of the target consumer and the latest message in the first message queue is greater than a preset message number threshold; or determining the consumption request based on the second message queue when a fault is found at the message source.

[0011] Optionally, the method further includes: the message source and the message destination verifying the integrity of the synchronized message; and the message destination repairing the synchronized message in the second message queue based on the first message queue if it determines that the synchronized message has failed the integrity verification.

[0012] Optionally, the method for determining the current consumption progress of the target consumer includes at least one of the following: determining the current consumption progress of the target consumer based on the consumption request sent by the target consumer; determining the current consumption progress of the target consumer based on the current consumption progress of different consumers stored at the consumption request processing end; and determining the current consumption progress of the target consumer based on the current consumption progress of different consumers for the first message queue recorded at the message source end.

[0013] A second aspect of this application provides another message processing method applied to a consumption request processing end; the consumption request processing end is used to: process consumption requests based on a first message queue at the message source end and a second message queue at the message destination end; the message source end is used to: synchronize messages in the first message queue to the second message queue; the method includes: for a consumption request of a target consumer, selecting whether to process the target consumption request based on the first message queue or the second message queue; if it is determined that the target consumption request is to be processed based on the second message queue, determining the target message corresponding to the current consumption progress of the target consumer in the second message queue, and processing the target consumption request based on messages following the target message in the second message queue.

[0014] A third aspect of this application provides a message processing system, comprising: a message source, a message destination, and a consumption request processing end; the message source is configured to: synchronize messages in a first message queue of the message source to a second message queue of the message destination; the consumption request processing end is configured to: select whether to process the consumption request based on the first message queue or the second message queue for a consumption request from a target consumer; if the consumption request is determined to be processed based on the second message queue, determine the target message in the second message queue corresponding to the current consumption progress of the target consumer, and process the consumption request based on messages following the target message in the second message queue.

[0015] A fourth aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps performed by the message source, message destination, or consumer request processing end in the above method.

[0016] The fifth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein when the computer program or instructions are executed by a processor, they implement the steps performed by the message source, message destination, or consumer request processing end in the above method.

[0017] A sixth aspect of this application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps performed by the message source, message destination, or consumer request processing end in the above method. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 This illustration schematically depicts an application scenario of a message processing method according to an embodiment of this application.

[0020] Figure 2 A flowchart illustrating a message processing method according to an embodiment of this application is shown schematically.

[0021] Figure 3 This schematically illustrates a structural block diagram of a message processing system according to an embodiment of the present application;

[0022] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a message processing method according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] Currently, message subscription mechanisms are widely used in business applications. In this mechanism, producers and consumers communicate asynchronously through message service nodes. Producers publish messages to the message service nodes, and consumers retrieve these messages from the nodes. Message service nodes can store and distribute messages using message topics or message queues. However, the stability of individual message service nodes is currently poor, leading to instability during message consumption for consumers. For example, if a single message service node fails or is under excessive load, consumers may be unable to retrieve messages, resulting in poor stability.

[0028] To address the aforementioned technical problems, embodiments of this application provide a message processing method.

[0029] In this method, a primary and backup system can be deployed, referred to as the message source and message destination, respectively. The message source and message destination can be message service nodes, or a system composed of multiple message service nodes, used to store messages and act as intermediaries between producers and consumers. As the primary and backup system, the message destination can serve as a backup system for the message source. In the event of a failure or excessive load at the message source, consumers can be reassigned to the message destination for message consumption, thus improving the stability of message consumption.

[0030] For a primary / standby system, messages from the message source can be synchronized to the message target. The specific message synchronization process is not limited. Specifically, the message source can directly send messages from its message queue to the message target and store them in the message queue of the message target. In this process, the consistency of message synchronization and the consistency of message order in the message queue can be ensured through integrity verification and other methods.

[0031] For messages synchronized in a primary / standby system, the storage space allocated to messages and the methods for updating historical messages may differ between the source and destination devices. Therefore, the message offset in the message queue at the source may differ from that at the destination. For example, the message offset at the source might be 500, indicating the 500th message in the queue; however, due to the larger number of historical messages stored at the destination, the offset might be 1000, indicating the 1000th message. The message offset serves as a unique identifier for precisely locating each message within the queue or partition. It can also be used as consumption progress, determining the consumer's consumption offset. For instance, if a consumer is currently consuming the 400th message, the message offset of 400 can be used as the consumption offset.

[0032] Accordingly, the mapping relationship between message offsets or consumption offsets can be determined in real time for the message source and message destination. For example, the message offset of the same message at the message source could be 500, and the message offset at the message destination could be 4000, thus determining the mapping relationship between 500 at the message source and 4000 at the message destination. Specifically, a message can be selected to determine its message offset at both the message source and message destination. Since the message order is the same between message queues, the mapping relationship between message offsets can be determined based on the difference between the message offsets of the same message at the message source and message destination, and thus, the mapping relationship of consumption offsets can also be determined. Of course, other methods can also be used to determine the mapping relationship, such as message timestamps or unique message identifiers, etc.

[0033] When consumers use a backup message destination for message consumption, it is necessary to determine the mapping relationship of consumption offsets to further determine the latest consumption progress of consumers at the message source. By combining the mapping relationship of consumption offsets, the latest consumption progress of consumers at the message destination can be determined. This allows message consumption to continue from the determined latest consumption progress, which can reduce message duplication or message omission and improve the consumer experience.

[0034] In addition, load balancing can be performed on the message source and message destination sides based on message synchronization. Consumer consumption requests can be processed based on the message source or message destination side, specifically based on the load status of the message source and message destination sides, thereby improving the degree of load balancing. This can improve the overall stability of message consumption, reduce resource waste, and improve resource utilization.

[0035] It should be noted that the message processing method and system provided in the embodiments of this application can be applied to the fields of distributed technology and fintech. For example, the message processing method provided in the embodiments of this application can be used to process message subscription mechanisms in the financial or banking fields. The message processing method provided in the embodiments of this application can also be applied to any field other than fintech, and the message subscription mechanism in any field can be processed using the message processing method provided in the embodiments of this application. The application fields of the message processing method and system provided in the embodiments of this application are not limited.

[0036] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0037] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0038] Figure 1 The illustration shows an application scenario diagram of a message processing method according to an embodiment of this application.

[0039] like Figure 1 As shown, application scenario 100 according to this embodiment may include: a first terminal device 101, a second terminal device 102, a third terminal device 103, and a server cluster 105. Network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server cluster 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0040] Users can use the first terminal device 101, the second terminal device 102, or the third terminal device 103 to interact with the server cluster 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0042] Server cluster 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, or the third terminal device 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0043] It should be noted that the message processing method provided in this application embodiment can generally be executed by server cluster 105. Correspondingly, the message processing system provided in this application embodiment can generally be set up in server cluster 105. The message processing method provided in this application embodiment can also be executed by a server cluster different from server cluster 105 but capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server cluster 105. Correspondingly, the message processing system provided in this application embodiment can also be set up in a server cluster different from server cluster 105 but capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server cluster 105.

[0044] In this embodiment, the terminal device can act as a consumer terminal and send a consumption request to the server cluster 105, so that the server cluster 105 can execute a message processing method provided in this application.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0046] Figure 2 A flowchart illustrating a message processing method according to an embodiment of this application is shown schematically.

[0047] like Figure 2 As shown, the message processing method provided in this embodiment may include operations S210 and S220. The embodiments of this application do not limit the executing entity of a message processing method. Optionally, it can be applied to a distributed system, where message synchronization operations are performed through a message source and a message destination in the distributed system. The distributed system may also include a consumer request processing end, which can be used to process consumer requests and can select between the message source and the message destination to process consumer requests.

[0048] Therefore, optionally, the message processing method can be applied to a pre-defined distributed system, which may include: a message source, a message destination, and a consumer request processing end.

[0049] In operation S210, the message source synchronizes the messages in the first message queue to the second message queue of the message destination.

[0050] In operation S220, the consumption request processing end selects to process the consumption request based on the first message queue or the second message queue for the consumption request of the target consumer; if it is determined to process the consumption request based on the second message queue, the current consumption progress of the target consumer is determined to be the target message in the second message queue, and the consumption request is processed based on the messages after the target message in the second message queue.

[0051] This method can be based on message synchronization between the message source and the message destination, making it convenient to select the message source or message destination to process the consumption request. This can improve the flexibility and stability of consumption request processing and enhance the stability of consumers when consuming messages.

[0052] The embodiments of this application do not limit the execution order between operations S210 and S220. Operations S210 and S220 can be executed in parallel, specifically, they can be executed continuously in parallel. Operation S220 can be based on message synchronization between the message source and the message destination. For example, operation S210 can be executed continuously to synchronize messages in the first message queue of the message source to the second message queue of the message destination in real time. Accordingly, consumption requests can be processed in parallel.

[0053] The embodiments of this application do not limit the first message queue and the second message queue. The first message queue can be a message queue in the message source end, specifically any message queue in the message source end. It is understood that the message source end can store one or more message queues, which can be synchronized to the message target end respectively. Different message queues in the message source end can correspond one-to-one with different message queues in the message target end. The message source end can synchronize messages in any message queue to the corresponding message queue in the message target end. For ease of description, any message queue in the message source end is referred to as the first message queue, and the message queue corresponding to the message target end is referred to as the second message queue. The first message queue can belong to any message topic, and correspondingly, the second message queue also belongs to the same message topic.

[0054] The embodiments of this application do not limit the message source and message destination. Optionally, the message source and message destination can be electronic devices, node clusters, or applications. In one specific embodiment, the message source can be a message service node cluster, the message destination can be another message service node cluster, and the message destination can serve as a backup system for the message source.

[0055] The embodiments of this application do not limit the consumption requests of the target consumer. Optionally, the target consumer can be any consumer. For ease of description, the consumer corresponding to any consumption request that needs to be processed is referred to as the target consumer, and message consumption services can be provided to the target consumer. Specifically, the target consumer can be a single consumer or a consumer group composed of multiple consumers, and different consumers in the consumer group can synchronize their consumption progress. Optionally, the consumption request can be used to represent that the target consumer requests to consume messages. Specifically, it can be to start message consumption as a new consumer or to continue message consumption based on previous consumption progress.

[0056] The embodiments of this application do not limit the specific content of the consumption request. Optionally, the consumption request may include information characterizing the current consumption progress of the target consumer, specifically the identifier information of the latest consumed message or the message offset of the latest consumed message. The consumption request may also include the message topic to be consumed, thereby facilitating the determination of the message queue to be consumed. Optionally, the target consumer's consumption request may specifically be a consumption request for the message topic to which the first message queue belongs, thereby determining whether the consumption request needs to be processed based on the first message queue or a second message queue that synchronizes messages in the first message queue. Alternatively, it may involve determining whether the target message queue is the first message queue or the second message queue for the target consumer's consumption request, and then further selecting the first message queue or the second message queue for processing.

[0057] The embodiments of this application do not limit the specific process of message synchronization between the message source and the message destination.

[0058] Optionally, the message source can directly send messages from the first message queue to the message destination. The message destination can directly store the received messages from the first message queue into the second message queue, achieving message synchronization. Alternatively, the message source can publish messages from the first message queue to a relay end through a message subscription mechanism. The message destination can subscribe to the messages published by the message source from the relay end, thereby obtaining the messages from the first message queue and storing them in the second message queue. The relay end can specifically be a consumption request processing end, facilitating the determination of the consumption offset mapping relationship or message offset mapping relationship between the message source and the message destination.

[0059] In addition, optionally, in order to improve message consistency and message queue consistency, integrity verification can be performed on synchronized messages. Specifically, integrity verification can be performed on the content of synchronized messages and the order between synchronized messages.

[0060] The embodiments of this application do not limit the specific integrity verification method. Specifically, a hash value can be used for verification. Specifically, the message source can generate a hash value for the synchronized message, which includes the message order of the synchronized message in the first message queue. Then, the generated hash value can be sent to the message destination. The message destination can generate a hash value for the received message, which includes the received message order, and compare it with the received hash value for verification, thereby verifying the integrity of the synchronized message content and message order.

[0061] The embodiments of this application do not limit the specific execution order of integrity verification. Specifically, the message destination can directly store the received message in the second message queue and then perform integrity verification; alternatively, the message destination can perform integrity verification first and then store the messages that pass integrity verification in the second message queue. Furthermore, for synchronization messages that fail integrity verification, further repair can be performed, specifically by requesting the message source to resend the synchronization message.

[0062] Therefore, optionally, the above method flow may further include: the message source and the message destination verifying the integrity of the synchronized messages; and the message destination, if it determines that the synchronized messages have failed the integrity verification, repairing the synchronized messages in the second message queue based on the first message queue. This embodiment can improve the consistency of message synchronization by performing integrity verification on the synchronized messages and repairing the synchronized messages that have failed the integrity verification.

[0063] The embodiments of this application do not limit the specific process for verifying the integrity of synchronized messages. Optionally, it may involve verifying the integrity of the message content and / or message order of synchronized messages, thereby improving the consistency of message content and message order.

[0064] The embodiments of this application do not limit the specific method of repair. Optionally, the message source can be requested to resynchronize the verified synchronized messages to the message target, so that the message target can perform repair; alternatively, the message source and the message target can compare the message content and message order of the synchronized messages stored in their own storage, locate the differences, and perform repair.

[0065] In one optional embodiment, the message target serves as a backup system for the message source. The message source can synchronize messages from the first message queue, specifically new messages, to the message target in real time to improve the real-time performance of message backup. Furthermore, the message target can perform integrity verification on the messages synchronized in real time. Only after the integrity verification passes can the messages be stored in the second message queue. It is understood that the message source can execute these two operations in parallel: message synchronization and message integrity verification. This improves the real-time performance of message synchronization and further enhances the consistency and accuracy of message synchronization.

[0066] Therefore, optionally, the message source synchronizes the messages in the first message queue to the second message queue of the message target. Specifically, this may include: the message source performing message synchronization and message verification operations in parallel; the message synchronization operation includes: synchronizing the messages in the first message queue of the message source to a preset storage area in the message target; the message verification operation includes: under a preset condition, generating integrity verification information for the message sequence to be verified in the first message queue that is not currently synchronized to the second message queue but is currently synchronized to the preset storage area, and the message order information in the message sequence to be verified, and sending the generated integrity verification information to the message target; if the message target determines, based on the received integrity verification information, that the message sequence to be verified and the message order information in the preset storage area have passed integrity verification, storing the message sequence to be verified in the preset storage area into the second message queue according to the message order information. This embodiment can improve the efficiency of message synchronization and message integrity verification by executing message synchronization and message verification operations in parallel, and can also improve the consistency and accuracy of message synchronization.

[0067] It is understood that the preset storage area in the message destination can serve as a buffer area to store messages synchronized from the message source for subsequent integrity verification. Specifically, the message source can synchronize messages from the first message queue to the preset storage area in the message destination in real time; this can specifically mean synchronizing newly added messages from the first message queue to the preset storage area in the message destination for synchronized updates. The embodiments of this application are not limited to a preset storage area; it can be any storage area in the message destination other than the second message queue.

[0068] The embodiments of this application are not limited to the preset situation. Optionally, the message source end can perform message synchronization operation and message verification operation in parallel. The message synchronization operation can be performed continuously in real time. The message synchronization operation is performed when there are messages that need to be synchronized. The message verification operation can be performed when a large number of messages have been synchronized. Specifically, it can be performed periodically or when a preset number of messages have been synchronized. Therefore, the preset situation can optionally include at least one of the following: (1) the duration between the current time and the last time the integrity verification was performed is greater than the preset duration; (2) the number of messages in the first message queue that have not been synchronized to the second message queue and have been synchronized to the preset storage area is greater than the preset number of messages.

[0069] The embodiments of this application do not limit the message sequence to be verified. It is understood that messages already synchronized to the preset storage area in the first message queue can be combined into a message sequence, specifically based on their order within the first message queue. Therefore, the message source can generate integrity verification information for the message sequence to be verified and the message order information within it. Specifically, this can be achieved by generating a hash value for the entire message sequence to be verified, the generation of which is related to the message order within the sequence; or by generating hash values ​​for each message content and its order information within the message sequence to be verified, facilitating subsequent location of discrepancies.

[0070] Correspondingly, messages received in the preset storage area can also be combined into a message sequence. Specifically, this combination can be based on the synchronized message order information, or on the timestamp information within the messages, etc. Therefore, the message destination can perform integrity verification on the message sequence to be verified in the preset storage area based on the integrity verification information generated by the message source. Specifically, it can generate integrity information for the message sequence to be verified in the preset storage area in the same way and compare it with the integrity verification information. The message destination can determine that the integrity verification has passed if the integrity information and integrity verification information are the same; otherwise, it determines that the integrity verification has failed.

[0071] Accordingly, if the integrity verification passes, the message target can store the sequence of messages to be verified in the preset storage area into the second message queue according to the message order information, which can improve the consistency of message content and message order during message synchronization. If the integrity verification fails, the message target can also perform a repair operation; for details, please refer to the explanations in other embodiments.

[0072] It is understood that, based on the explanation of the above embodiments, the efficiency and consistency of message synchronization can be improved. Specifically, the above message synchronization process can be continuously executed in real time, improving the consistency and real-time performance of message synchronization between the message source and the message destination. This makes it convenient for the message destination to act as a backup system for the message source to process consumption requests, and also makes it convenient for the second message queue to act as a backup message queue for the first message queue to process consumption requests.

[0073] The embodiments of this application do not limit the specific process of the consumer request processing end in processing consumer requests.

[0074] Optionally, based on the first message queue and the second message queue for message synchronization, the consumption request can be processed first based on the first message queue or the second message queue, and then the specific consumption request processing flow can be further executed.

[0075] In one optional embodiment, the first message queue may be a message queue in the message source end, and the second message queue may be a message queue in the message destination end. Thus, for a consumption request, the first message queue corresponding to the message topic requested by the consumption request in the message source end, or the second message queue corresponding to the message topic requested by the consumption request in the message destination end, can be selected.

[0076] Optionally, a consumption request can also be a request to consume a message topic belonging to the first message queue, thereby determining whether to select the first message queue or the second message queue for processing.

[0077] The embodiments of this application do not limit the specific method of selecting the first message queue or the second message queue. Optionally, the selection can be made by load balancing, or based on whether the message source and message destination are faulty, or based on the load of the message source and message destination and network communication conditions (e.g., network communication rate), etc.

[0078] Optionally, selecting whether to process a consumption request based on a first message queue or a second message queue may include any of the following: (1) determining a consumption request based on a second message queue when the number of consumers targeting the first message queue is greater than a preset consumer number threshold; (2) determining a consumption request based on a second message queue when the number of messages between the current consumption progress of the target consumer and the latest message in the first message queue is greater than a preset message number threshold; (3) determining a consumption request based on a second message queue when a fault is found at the message source. This embodiment, by limiting the specific selection method, can select the second message queue to process consumption requests when the first message queue is overloaded, or when there are messages that have been consumed for a relatively long time, or when there is a fault at the message source, thereby sharing the load of the first message queue and improving the stability of consumers when consuming messages.

[0079] Understandably, in other cases, it can be determined that the consumption request is based on the processing of the first message queue.

[0080] The embodiments of this application are not limited to the process of processing consumption requests based on the second message queue. Optionally, the current consumption progress of the target consumer can be determined by identifying the target message in the second message queue, and the consumption request can be processed based on the messages following the target message in the second message queue. Specifically, consumption can start from the target message.

[0081] The embodiments of this application are not limited to the process of processing consumption requests based on the first message queue. Optionally, the current consumption progress of the target consumer can be determined by identifying the target message in the first message queue, and the consumption request can be processed based on messages following the target message in the first message queue. Specifically, consumption can start from the target message. For a detailed explanation, please refer to the explanation of the second message queue in other embodiments.

[0082] The embodiments of this application do not limit the specific method of processing consumption requests. Optionally, processing the consumption request based on the messages following the target message in the second message queue can specifically involve the consumption request processing end acting as a relay, sending the required message to the target consumer based on the messages following the target message in the second message queue according to the needs of the consumption request; alternatively, the message target end can establish a communication connection with the target consumer, thereby allowing the message target end to further send the required message to the target consumer based on the messages following the target message in the second message queue according to the needs of the consumption request.

[0083] Accordingly, the consumption request targeted by message processing based on the target message in the first message queue can be specifically handled by the consumption request processing end acting as a relay, sending the required message to the target consumer based on the message following the target message in the first message queue according to the needs of the consumption request; or the message source end can establish a communication connection with the target consumer, so that the message source end can further send the required message to the target consumer based on the message following the target message in the first message queue according to the needs of the consumption request.

[0084] The embodiments of this application do not limit the specific method for determining the current consumption progress of the target consumer. Optionally, the current consumption progress can be determined based on the target consumer's historical consumption history. For new consumers, the current consumption progress can be determined based on specific consumption needs, specifically by determining the message corresponding to a specified time or the first message in the message queue. For consumers who have already consumed, the current consumption progress can be determined based on the previous consumption situation. Specifically, the current consumption progress of each consumer can be stored at the message source, message destination, or consumption request processing end, or the current consumption progress can be provided by the consumer.

[0085] Optionally, the method for determining the current consumption progress of the target consumer includes at least one of the following: (1) determining the current consumption progress of the target consumer based on the consumption request sent by the target consumer; (2) determining the current consumption progress of the target consumer based on the current consumption progress of different consumers stored at the consumption request processing end; (3) determining the current consumption progress of the target consumer based on the current consumption progress of different consumers for the first message queue recorded at the message source end. This embodiment can determine the current consumption progress of the consumer through multiple information sources, which can improve the accuracy of the current consumption progress of the consumer.

[0086] Understandably, target consumers can determine their current consumption progress and add it to their consumption requests, making it easier to determine the current progress based on the requests. The consumption request processor or message source can store the current consumption progress of each consumer for easy querying.

[0087] The embodiments of this application do not limit the specific form of the target consumer's current consumption progress. Specifically, it can be information such as the identifier of the latest message currently consumed, message offset, or timestamp. This information can be used to represent the latest message consumed by the target consumer in the first message queue, or it can directly represent the latest message consumed by the target consumer.

[0088] In one optional embodiment, the target message is determined based on the second message queue. This can be done by determining the target message in the second message queue corresponding to the current consumption progress of the target consumer, based on the message offset mapping relationship between the first and second message queues, that is, the message offset mapping relationship of the same message in the first and second message queues.

[0089] Therefore, optionally, the target consumer's current consumption progress may include: the latest message consumed by the target consumer in the first message queue; determining the target message corresponding to the target consumer's current consumption progress in the second message queue may specifically include: based on the offset mapping relationship of the same message between the first and second message queues, determining the message corresponding to the latest message consumed by the target consumer in the first message queue in the second message queue as the target message. This embodiment can combine the offset mapping relationship of the same message between the first and second message queues to determine the target message, which can improve the accuracy and efficiency of determining the target message.

[0090] Optionally, the offset mapping relationship for the same message between the first message queue and the second message queue can be predetermined or determined in real time. Determining the offset mapping relationship in real time can improve its accuracy and timeliness.

[0091] Therefore, optionally, when determining the consumption request to be processed based on the second message queue, the above method flow may further include: the consumption request processing end determining the offset mapping relationship of the same message between the first message queue and the second message queue in real time. This embodiment can determine the offset mapping relationship in real time when determining the consumption request to be processed based on the second message queue, which can improve the accuracy and real-time performance of the offset mapping relationship and improve the accuracy of determining the target message.

[0092] Optionally, the offset mapping relationship of the same message between the first message queue and the second message queue can be predetermined and stored. Then, the target message can be determined based on the pre-stored offset mapping relationship. Furthermore, the determined target message can be compared with the current consumption progress of the target consumer to verify the target message. For example, if the determined target message differs from the current consumption progress of the target consumer (i.e., it differs from the latest message consumed by the target consumer in the first message queue), it can be determined that there is a deviation in the pre-determined offset mapping relationship. Therefore, the offset mapping relationship of the same message between the first message queue and the second message queue can be determined in real time, and then the target message can be determined.

[0093] The embodiments of this application do not limit the specific method for determining the offset mapping relationship of the same message between the first message queue and the second message queue. Optionally, the message offset can be determined separately for the same message in the first message queue and the second message queue, thereby constructing an offset mapping relationship; alternatively, the message offset can be determined separately for messages in the first message queue and the second message queue with the same timestamp or the same unique message identifier, thereby constructing an offset mapping relationship. It is understood that, through the above methods, the offset mapping relationship of the same message between the first message queue and the second message queue can be determined in real time.

[0094] In one optional embodiment, the target message is determined based on the second message queue. This can be achieved by directly identifying the corresponding target message in the second message queue based on the message identifier recently consumed by the target consumer, which can improve the accuracy and efficiency of determining the target message.

[0095] Optionally, the target consumer's current consumption progress may include: the identifier information of the target consumer's latest consumed message; determining the target message corresponding to the target consumer's current consumption progress in the second message queue, specifically, may include: identifying the message corresponding to the identifier information of the target consumer's latest consumed message in the second message queue as the target message. This embodiment can determine the corresponding target message in the second message queue based on the identifier information of the target consumer's latest consumed message, which can improve the accuracy and efficiency of determining the target message.

[0096] It is understandable that the corresponding message can be queried in the second message queue based on the message's identifier information, thereby easily identifying the target message for subsequent processing of consumption requests. Correspondingly, the target message can also be identified based on the first message queue using the method described in the above embodiment.

[0097] For ease of understanding, embodiments of this application also provide a one-sided message processing method embodiment, which can be applied to a consumer request processing end. Specifically, a message processing method can be applied to a consumer request processing end; the consumer request processing end is used to: process consumer requests based on a first message queue at the message source end and a second message queue at the message destination end; the message source end is used to: synchronize messages in the first message queue to the second message queue; the method may include: for a consumer request from a target consumer, selecting whether to process the target consumer request based on the first message queue or the second message queue; if it is determined that the target consumer's current consumption progress should be processed based on the second message queue, determining the target message in the second message queue corresponding to the target consumer's current consumption progress, and processing the target consumer request based on messages following the target message in the second message queue. This method flow can be based on message synchronization between the message source end and the message destination end, facilitating the selection of the message source end or the message destination end to process the consumer request, improving the flexibility and stability of consumer request processing, and enhancing the stability of message consumption by the consumer. For an explanation of this embodiment, please refer to the explanations of other embodiments.

[0098] Optionally, the message synchronization method between the message source and the message target includes: the message source performs message synchronization and message verification operations in parallel; the message synchronization operation includes: synchronizing messages in the first message queue of the message source to a preset storage area in the message target; the message verification operation includes: under a preset condition, generating integrity verification information for the message sequence to be verified in the first message queue that has not yet been synchronized to the second message queue but has now been synchronized to the preset storage area, and the message order information in the message sequence to be verified, and sending the generated integrity verification information to the message target; if the message target determines that the message sequence to be verified and the message order information in the preset storage area have passed integrity verification based on the received integrity verification information, it stores the message sequence to be verified in the preset storage area into the second message queue according to the message order information.

[0099] Optionally, the message source and message destination are used to: verify the integrity of the synchronized message; the message destination is used to: repair the synchronized message in the second message queue based on the first message queue if it is determined that the synchronized message has failed the integrity verification.

[0100] Optionally, the current consumption progress of the target consumer can be determined by at least one of the following methods: (1) determining the current consumption progress of the target consumer based on the consumption request sent by the target consumer; (2) determining the current consumption progress of the target consumer based on the current consumption progress of different consumers stored at the consumption request processing end; (3) determining the current consumption progress of the target consumer based on the current consumption progress of different consumers for the first message queue recorded at the message source end.

[0101] For ease of understanding, this application also provides an application embodiment. In this embodiment, the following concepts are explained first: A service node, specifically a message middleware server node, is responsible for receiving and storing messages and processing service client requests. A source / destination end, specifically the starting point (source) and ending point (destination) of the message flow, is a distributed system composed of multiple service nodes. The source end can synchronize its own messages to the destination end, allowing the destination end to serve as a backup system. A message topic, specifically a logical unit on the message engine that stores different types of messages, used to distinguish specific business logic. When a producer produces a message, it specifies the topic to which the message belongs; when a consumer consumes a message, it specifies the topic and retrieves the messages produced by the producer within that topic.

[0102] Consumer group offset can be seen as a record of a consumer group's consumption progress within a message topic partition. A consumer group can be a collection of multiple consumers, who can share consumption progress. Essentially, a consumer group can be viewed as a special collective of consumers. Specifically, the consumption progress can be the offset of the message currently being consumed by the consumer within the message queue of the message topic partition.

[0103] Offset mapping can be seen as a transformation relationship between the offsets of consumer groups between the source and target ends. It can be understood that the message offset of the same message in the source end and the consumption offset in the target end can differ between the synchronized message queues. Therefore, a transformation or mapping relationship can be deployed to easily locate the message's position in the source and target ends, thereby facilitating the determination of the current consumption progress or offset of the consumer group in both ends. Since the messages in the synchronized message queues are in the same order, the transformation relationship can be determined based on the difference in message offsets.

[0104] The purpose of this embodiment is to complete cross-terminal lossless transmission of massive historical messages without interrupting business operations, improve message consistency, and ensure that critical systems such as financial transactions achieve the migration goal of zero message loss and strong consistency of consumption status under continuous service, while also ensuring displacement consistency in similar scenarios such as disaster recovery switching.

[0105] This embodiment deploys an embedded optimized data replicator, constructs a displacement mapping table (containing core fields such as source identifier, consumer group, topic, partition, and source / target displacement value) through a metadata database, and introduces the following innovative mechanisms.

[0106] 1. Disaster Recovery Storage and Integrity Verification Mechanism: The source end synchronizes messages to the target end's independent disaster recovery storage area (non-consumption queue) in real time, achieving real-time backup of message content. The source end periodically (e.g., every 5 minutes) generates an integrity digest (using hash values ​​to ensure the message order cannot be tampered with) for the synchronized message sequence and sends the digest to the target end. After the target end verifies the digest, it appends the messages in the disaster recovery storage area to the consumption queue in order, forming a closed-loop process of "disaster recovery storage → digest verification → queue append".

[0107] 2. Consumption routing strategy based on load balancing: The conditions for consumer groups to select target end for consumption are limited to load balancing scenarios. Deploy intelligent traffic splitting controllers to provide multiple traffic splitting methods. (1) Topic pressure detection: If the number of consumers in a single topic exceeds the threshold (e.g., 64), then new consumption requests for that topic will be routed to the target end. (2) Consumption progress splitting: When a consumer requests to consume historical messages (e.g., with a low offset), it will be routed to the target end; when a consumer requests to consume real-time messages (e.g., with a high offset), it will be routed to the source end.

[0108] 3. Consumption offset mapping scheme: Message location can be achieved using timestamps and unique business keys to determine the corresponding message offsets in the target and source ends, allowing for real-time construction of consumption offset mapping relationships. Alternatively, a message identifier location method can be used. When routing consumption requests, the load balancer captures the latest consumption message identifier of the consumer at the source end and stores it in the metadata database. When the consumer group switches to the target end, the offset translator queries the latest consumption message identifier based on the consumer group identifier and scans for a matching position in the target end's disaster recovery storage area as the consumption starting point.

[0109] Furthermore, the displacement translator can capture the consumption data from the source end in real time and persist it to the displacement information table in the database within milliseconds. Simultaneously, a dynamic mapping maintenance mechanism is deployed: based on the message data recorded in the logs (including message data from all topics within 15 minutes from both the source and target ends), it iterates and compares the message differences for the same topic from both ends, replays the missing interval messages, and atomically updates the database mapping records, achieving self-healing of displacement deviations through sub-second periodic verification.

[0110] The operation steps performed in this embodiment include the following 7 steps.

[0111] Step 1: When the consumer group starts consuming on the target end, the displacement translator intercepts the consumer group displacement synchronization request, parses the consumer group identifier, topic and partition information, and queries the displacement mapping table in the metadata database.

[0112] Step 2: If the target displacement record for the consumer group exists in the mapping table, proceed directly to step 4; otherwise, proceed to step 3.

[0113] Step 3: Trigger the displacement translator to query the latest consumed message identifier stored in the metadata database based on the consumer group identifier, scan the message position matching the identifier in the message queue on the target end, generate a displacement mapping record and persist it to the metadata database, and then execute Step 4.

[0114] Step 4: The displacement translator dynamically compares the displacement value of the source end consumption displacement with the consumption progress of the target end by capturing it in real time, and then executes Step 5.

[0115] Step 5: If a deviation in the source / target displacement value is detected, step 6 is triggered immediately.

[0116] Step 6: The displacement translator writes the mapped target displacement value to the target end, completing the consumer group displacement synchronization. Consumers then begin consuming messages from the target end.

[0117] Step 7: The mapping corrector replays the missing message intervals based on the message synchronization log, repairs the data gaps on the target end, updates the mapping records in the metadata database, and periodically performs displacement verification to achieve self-healing of displacement deviations.

[0118] This embodiment uses a dynamic displacement mapping mechanism to determine the consumption displacement mapping relationship in real time, enabling production topics containing massive amounts of existing data (such as financial transaction records) to achieve lossless migration across clusters with zero business interruption, ensuring zero message loss and strong consistency of consumption status.

[0119] The new technical improvements offer the following core advantages: the target-side disaster recovery storage area is independent of the consumption queue, reducing displacement chaos caused by accidental writes. Intelligent traffic distribution based on topic pressure and consumption progress reduces the load on the source end. The message identification-based location method improves the efficiency of message location.

[0120] Corresponding to the above method embodiments, this application also provides a system embodiment. The following will be combined with... Figure 3 The system is described in detail.

[0121] Figure 3 The diagram illustrates a structural block diagram of a message processing system according to an embodiment of this application.

[0122] like Figure 3 As shown, the message processing system 300 provided in this embodiment may include: a message source end 310, a message destination end 320, and a consumption request processing end 330.

[0123] The message source end 310 can be used to synchronize messages in the first message queue of the message source end to the second message queue of the message destination end. In one embodiment, the message source end 310 can be used to perform operation S210 and related operations.

[0124] The consumption request processing terminal 330 can be used to: select whether to process the consumption request based on a first message queue or a second message queue for a given consumption request from a target consumer; if it is determined that the consumption request should be processed based on the second message queue, determine the target message corresponding to the current consumption progress of the target consumer in the second message queue, and process the consumption request based on messages following the target message in the second message queue. In one embodiment, the consumption request processing terminal 330 can be used to execute operation S220 and related operations.

[0125] Optionally, the message target terminal 320 can be used to perform the operations performed by the message target terminal in the above method embodiments.

[0126] Optionally, the current consumption progress of the target consumer includes: the latest message consumed by the target consumer in the first message queue; the consumption request processing terminal 330 is used to: determine the message corresponding to the latest message consumed by the target consumer in the first message queue in the second message queue as the target message based on the offset mapping relationship of the same message between the first message queue and the second message queue.

[0127] Optionally, the current consumption progress of the target consumer includes: the identification information of the latest consumed message of the target consumer; the consumption request processing terminal 330 is used to: determine the message corresponding to the identification information of the latest consumed message of the target consumer in the second message queue as the target message.

[0128] Optionally, the consumption request processing end 330 is further configured to: determine in real time the offset mapping relationship of the same message between the first message queue and the second message queue when it is determined that the consumption request is to be processed based on the second message queue.

[0129] Optionally, the message source end 310 is used to perform message synchronization operation and message verification operation in parallel; the message synchronization operation includes: synchronizing messages in the first message queue of the message source end to a preset storage area in the message target end; the message verification operation includes: under a preset condition, generating integrity verification information for the message sequence to be verified in the first message queue that has not yet been synchronized to the second message queue but has now been synchronized to the preset storage area, and the message order information in the message sequence to be verified, and sending the generated integrity verification information to the message target end; the message target end 320 is used to, based on the received integrity verification information, determine that the message sequence to be verified in the preset storage area and the message order information have passed integrity verification, and then store the message sequence to be verified in the preset storage area into the second message queue according to the message order information.

[0130] Optionally, the consumption request processing terminal 330 is configured to perform any of the following: if it is determined that the number of consumers for the first message queue is greater than a preset consumer number threshold, determine a consumption request for processing based on the second message queue; if it is determined that the number of messages between the current consumption progress of the target consumer and the latest message in the first message queue is greater than a preset message number threshold, determine a consumption request for processing based on the second message queue; if it is determined that there is a fault at the message source end, determine a consumption request for processing based on the second message queue.

[0131] Optionally, the message source end 310 and the message destination end 320 are used to verify the integrity of the synchronized message; the message destination end 320 is used to repair the synchronized message in the second message queue based on the first message queue if it is determined that the synchronized message has failed the integrity verification.

[0132] Optionally, the method for determining the current consumption progress of the target consumer includes at least one of the following: determining the current consumption progress of the target consumer based on the consumption request sent by the target consumer; determining the current consumption progress of the target consumer based on the current consumption progress of different consumers stored at the consumption request processing end; and determining the current consumption progress of the target consumer based on the current consumption progress of different consumers for the first message queue recorded at the message source end. Specifically, the consumption request processing end 330 may be used to determine the current consumption progress of the target consumer by executing the above-mentioned method for determining the current consumption progress of the target consumer.

[0133] For an explanation of this system embodiment, please refer to the explanations of other embodiments. It is understood that any operation in the above method embodiments can be executed by the corresponding device in this system embodiment.

[0134] Corresponding to the above-described unilateral method embodiments, embodiments of this application also provide a message processing apparatus, which can be applied to a consumption request processing end; the consumption request processing end is used to: process consumption requests based on a first message queue at the message source end and a second message queue at the message target end; the message source end is used to: synchronize messages in the first message queue to the second message queue; the apparatus includes: a selection module, used to select whether to process the consumption request based on the first message queue or the second message queue for the consumption request of the target consumer; a processing module, used to, when determining that the consumption request is to be processed based on the second message queue, determine the target message corresponding to the current consumption progress of the target consumer in the second message queue, and process the consumption request based on messages after the target message in the second message queue.

[0135] For further explanation of this device embodiment, please refer to other embodiments. It is understood that the operations performed by the consumer request processing terminal in the above method embodiment can be performed by any unit in this device embodiment.

[0136] According to embodiments of this application, any multiple modules in the selection module and processing module can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the selection module and processing module can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the selection module and processing module can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0137] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a message processing method according to an embodiment of this application.

[0138] like Figure 4As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0139] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in the one or more memories, specifically executing steps performed by the message source, message destination, or consumer request processing end in any of the above method embodiments.

[0140] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0141] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The aforementioned computer-readable storage medium carries one or more programs, which, when executed, implement the steps performed by the message source, message destination, or consumer request processing end in the method according to the embodiments of this application.

[0142] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0143] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the steps executed by the message source, message destination, or consumer request processing end in any of the method embodiments provided in this application.

[0144] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0145] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0146] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0147] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A message processing method, characterized in that, Applied to pre-defined distributed systems; The pre-defined distributed system includes: a message source, a message destination, and a consumer request processing end; the method includes: The message source end synchronizes the messages in the first message queue to the second message queue of the message target end; For a consumption request from a target consumer, the consumption request processing terminal selects to process the consumption request based on either the first message queue or the second message queue. If it is determined that the consumption request should be processed based on the second message queue, the terminal determines the target message in the second message queue corresponding to the current consumption progress of the target consumer, and processes the consumption request based on the messages following the target message in the second message queue.

2. The method according to claim 1, characterized in that, The current consumption progress of the target consumer includes: the latest message consumed by the target consumer in the first message queue; Determining the target message in the second message queue corresponding to the current consumption progress of the target consumer includes: Based on the offset mapping relationship of the same message between the first message queue and the second message queue, the message in the second message queue corresponding to the latest message consumed by the target consumer in the first message queue is determined as the target message.

3. The method according to claim 1, characterized in that, The current consumption progress of the target consumer includes: the identification information of the latest consumption message of the target consumer; Determining the target message in the second message queue corresponding to the current consumption progress of the target consumer includes: The message corresponding to the identifier information of the latest message consumed by the target consumer in the second message queue is identified as the target message.

4. The method according to claim 2, characterized in that, When determining the consumption request to be processed based on the second message queue, the method further includes: the consumption request processing end determining in real time the offset mapping relationship of the same message between the first message queue and the second message queue.

5. The method according to claim 1, characterized in that, The message source synchronizes messages from the first message queue to the second message queue of the message destination, including: The message source performs message synchronization and message verification operations in parallel. The message synchronization operation includes: synchronizing messages in the first message queue at the message source end to a preset storage area at the message target end; The message verification operation includes: under a preset condition, generating integrity verification information for a sequence of messages to be verified in the first message queue that is not currently synchronized to the second message queue but is currently synchronized to the preset storage area, and the message order information in the sequence of messages to be verified, and sending the generated integrity verification information to the message target end; If the message target terminal determines that the message sequence to be verified in the preset storage area and the message order information have passed the integrity verification based on the received integrity verification information, it stores the message sequence to be verified in the preset storage area into the second message queue according to the message order information.

6. The method according to claim 1, characterized in that, The selection is based on the consumption request targeted by the first message queue or the second message queue, including any one of the following: If it is determined that the number of consumers targeting the first message queue is greater than a preset consumer number threshold, then the consumption request to be processed is determined based on the second message queue. If the number of messages between the current consumption progress of the target consumer and the latest message in the first message queue is greater than a preset message number threshold, the consumption request to be processed based on the second message queue is determined. If a fault is found at the message source, the consumption request to be processed based on the second message queue is determined.

7. The method according to claim 1, characterized in that, The method further includes: The message source and the message destination verify the integrity of the synchronized message; If the target message determines that the synchronized message has failed integrity verification, it repairs the synchronized message in the second message queue based on the first message queue.

8. The method according to claim 1, characterized in that, The method for determining the current consumption progress of the target consumer includes at least one of the following: Based on the consumption request sent by the target consumer, determine the current consumption progress of the target consumer; Based on the current consumption progress of different consumers stored in the consumption request processing terminal, the current consumption progress of the target consumer is determined. Based on the current consumption progress of different consumers in the first message queue recorded at the message source, the current consumption progress of the target consumer is determined.

9. A message processing method, characterized in that, It is applied to the consumption request processing end; the consumption request processing end is used to: process consumption requests based on the first message queue at the message source end and the second message queue at the message destination end; the message source end is used to: synchronize the messages in the first message queue to the second message queue; The method includes: For a consumption request from a target consumer, the consumer request is processed based on either the first message queue or the second message queue. If a consumption request is determined based on the second message queue, the current consumption progress of the target consumer is determined in the second message queue corresponding to the target message, and the consumption request is processed based on the messages following the target message in the second message queue.

10. A message processing system, characterized in that, include: Message source, message destination, and consumption request processing end; The message source end is used to: synchronize the messages in the first message queue of the message source end to the second message queue of the message target end; The consumption request processing terminal is used to: select whether to process the consumption request based on the first message queue or the second message queue for the consumption request of the target consumer; if it is determined that the consumption request should be processed based on the second message queue, determine the target message corresponding to the current consumption progress of the target consumer in the second message queue, and process the consumption request based on the messages after the target message in the second message queue.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic is that the one or more processors execute the one or more computer programs to implement the steps performed by the message source end, message destination end, or consumer request processing end in the method according to any one of claims 1 to 8, or to implement the steps of the method according to claim 9.

12. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps performed by the message source, message destination, or consumer request processing end in the method according to any one of claims 1 to 8, or implement the steps of the method according to claim 9.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps performed by the message source, message destination, or consumer request processing end in the method according to any one of claims 1 to 8, or implement the steps of the method according to claim 9.

Images