Message processing method, system and storage medium

Abstract

The application provides a message processing method, system and storage medium, and relates to the technical field of data processing. The method is applied to a source side service synchronization device of a message sending end; the method comprises the following steps: receiving a data change request sent by a client; the data change request comprises data identification and business change data; according to the data change request, a cache check delay message is sent to a distributed message cluster; the cache check delay message is used to trigger consistency check of a local cache unit in the source side service synchronization device when a time experienced since sending reaches a target synchronization delay; the business change data is written into a local storage node corresponding to the source side service synchronization device; and in the case of writing completion, a synchronization message is sent to the distributed message cluster; the synchronization message at least comprises data identification, business change data and a sending time stamp, and is used to instruct a target side service synchronization device of a message receiving end to update a corresponding local cache unit.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a message processing method, system and storage medium. Background Technology

[0002] With the rapid development of the domestic and international civil aviation markets and the explosive growth of flight volume, increasingly higher requirements are being placed on the stability, reliability, high throughput, high-speed transmission, proximity access, and accurate synchronization of the real-time civil aviation revenue assessment system and the flight dynamic pricing system.

[0003] In related technologies, traditional synchronization systems face a massive number of message synchronization requests at the same time, requiring second-level synchronization latency, rapid calculation and distribution. If a centralized forwarding synchronization system technology and architecture design are still used, it will lead to congestion of the message synchronization link, client response timeouts, and serious impact on business efficiency. Summary of the Invention

[0004] This application provides a message processing method, system, and storage medium for improving business response efficiency.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a message processing method is provided, applied to a source-side service synchronization device at the message sending end. The method includes: receiving a data change request sent by a client; the data change request includes a data identifier and business change data; sending a cache check delay message to a distributed message cluster based on the data change request; the cache check delay message triggers consistency verification of the local cache unit in the source-side service synchronization device when the time elapsed since sending reaches the target synchronization delay; writing the business change data to the corresponding local storage node of the source-side service synchronization device; and sending a synchronization message to the distributed message cluster upon completion of the writing process; the synchronization message includes at least a data identifier, business change data, and a sending timestamp, used to instruct the target-side service synchronization device at the message receiving end to update the corresponding local cache unit.

[0006] In one possible implementation, the target synchronization delay is determined by: obtaining a network delay estimate between the source-side service synchronization device and the target-side service synchronization device; obtaining the current load status index of the source-side service synchronization device; and obtaining the target synchronization delay based on the network delay estimate and the load status index.

[0007] In one possible implementation, the target synchronization delay is obtained based on network latency estimation and load status indicators, including: determining the corresponding processing time based on load status indicators; summing the network latency estimation and processing time to obtain a baseline delay; and obtaining the target synchronization delay based on the baseline delay and a preset redundancy coefficient. The preset redundancy coefficient is used to characterize the system's fault tolerance adjustment capability.

[0008] In one possible implementation, the method further includes: if the time elapsed since the self-caching check delay message is sent meets the target synchronization delay, obtaining the corresponding first data in the local cache unit of the source-side service synchronization device according to the data identifier; the first data is the current cached data of the local cache unit; obtaining the second data corresponding to the data identifier from the local storage node; the second data is used to represent the standard data source; performing a consistency comparison between the first data and the second data to obtain a comparison verification result; if the comparison verification result is inconsistent, updating the local cache unit based on the second data.

[0009] In one possible implementation, if the comparison and verification results are inconsistent, the method further includes: generating an abnormal synchronization message corresponding to the data identifier; sending the abnormal synchronization message to the distributed message cluster; wherein the abnormal synchronization message is used to instruct the target-side service synchronization device to set the data corresponding to the data identifier in the corresponding local cache unit to an invalid state, and to send a data recovery request to the source-side service synchronization device according to the data identifier.

[0010] In one possible implementation, the method further includes: receiving a data read request sent by a client; the data read request includes an identifier of the data to be read; querying the local cache unit of the source-side service synchronization device based on the identifier of the data to be read to obtain the cache hit status; if the cache hit status is a miss status, retrieving the business data corresponding to the identifier of the data to be read from the local storage node and writing the business data into the local cache unit.

[0011] Secondly, a message processing method is provided, applied to a target-side service synchronization device at a message receiving end; the method includes: receiving a synchronization message from a distributed message cluster; the synchronization message is sent by the source-side service synchronization device at the message sending end after successfully writing the data change request sent by the client into the local storage node; the synchronization message includes at least a data identifier, business change data, and a sending timestamp; and updating the local cache unit of the target-side service synchronization device according to the data identifier and the business change data.

[0012] In one possible implementation, the local cache unit corresponding to the target-side service synchronization device is updated based on the data identifier and the service change data. This includes: obtaining the receiving timestamp when the synchronization message is received; determining the current transmission delay based on the receiving timestamp and the sending timestamp; and writing the service change data into the local cache unit if the current transmission delay does not exceed the target synchronization delay. The target synchronization delay is the delay estimated by the source-side service synchronization device based on load status indicators and network latency.

[0013] In one possible implementation, writing business change data to a local cache unit includes: obtaining the second timestamp of historical data corresponding to the data identifier in the local cache unit; determining the target data content to be written to the local cache unit based on the sending timestamp, the second timestamp, and preset writing rules; and writing the target data content to the local cache unit.

[0014] In one possible implementation, the target data content to be written to the local cache unit is determined based on the sending timestamp, the second timestamp, and the preset writing rules, including: if the sending timestamp is later than the second timestamp, the business change data is determined as the target data content according to the preset writing rules.

[0015] In one possible implementation, the target data content to be written to the local cache unit is determined based on the sending timestamp, the second timestamp, and the preset writing rules. This includes: when the sending timestamp and the second timestamp are the same, determining the target data content to be written to the local cache unit based on the preset writing rules; wherein the preset writing rules are used to determine the target data content based on the priority of the data source.

[0016] In one possible implementation, the method further includes: receiving an abnormal synchronization message from a distributed message cluster; setting the data corresponding to the data identifier in the local cache unit to an invalid state according to the abnormal synchronization message; and in response to the invalid state, sending a data recovery request to the source-side service synchronization device according to the data identifier.

[0017] The technical effects of any implementation method in the second aspect can be found in the technical effects of any implementation method in the first aspect mentioned above, and will not be repeated here.

[0018] Thirdly, a message processing system is provided, which is deployed in multiple data centers. Each data center includes: a local storage node for storing business messages of the data center; a distributed message cluster for transmitting messages between the multiple data centers; a message synchronization component connected to the local storage node and the distributed message cluster for retrieving messages from the local storage node and sending them to other data centers in the multiple data centers via the distributed message cluster; and a component for receiving messages from other data centers via the distributed message cluster and writing them to the local storage node; and multiple service synchronization devices that are communicatively connected to the local storage node, and each service synchronization device is configured with a local cache unit.

[0019] Among them, multiple service synchronization devices are configured to: when acting as message senders, execute any of the methods in the first aspect; and when acting as message receivers, execute any of the methods in the second aspect.

[0020] Fourthly, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the method described above.

[0021] Fifthly, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, and the at least one computer program is loaded and executed by a processor to implement the methods described above.

[0022] Sixthly, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the methods described above.

[0023] The solutions provided in aspects three through six above are used to implement the method provided in aspect one above, and their specific implementations will not be described in detail here. The technical effects corresponding to any implementation method of the solutions provided in aspects two through six above can be found in the technical effects corresponding to any implementation method in aspect one above, and will not be described in detail here.

[0024] It should be noted that any of the possible implementations of any of the above aspects can be combined, provided that the solutions do not contradict each other.

[0025] The technical solution provided in this application brings at least the following benefits: By receiving data change requests and immediately sending cache check delay messages to the distributed message cluster, the trigger point for consistency verification can be pre-set before data writing, avoiding business errors caused by inconsistencies between cache and storage node data due to synchronization delays or anomalies. Secondly, sending synchronization messages after writing business change data to local storage nodes ensures the reliability of data persistence and prevents dirty reads caused by messages being delivered before data is written to the database. Finally, the synchronization message contains data identifiers, business change data, and sending timestamps, enabling the target side to accurately determine the update order based on the timestamps and complete local cache refresh, thereby achieving eventual consistency between the source and target caches. The entire process ensures the accuracy and fault tolerance of data synchronization while avoiding network overhead caused by frequent cross-node queries, thus significantly improving the overall system throughput and response speed, and ultimately effectively improving business efficiency. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the structure of a message processing system provided in an embodiment of this application; Figure 2 This is a schematic diagram of another message processing system provided in an embodiment of this application; Figure 3 A flowchart illustrating a message processing method provided in an embodiment of this application; Figure 4 A flowchart illustrating another message processing method provided in an embodiment of this application; Figure 5 A flowchart illustrating another message processing method provided in an embodiment of this application; Figure 6 A flowchart illustrating another message processing method provided in an embodiment of this application; Figure 7 A flowchart illustrating another message processing method provided in an embodiment of this application; Figure 8 A flowchart illustrating another message processing method provided in an embodiment of this application; Figure 9 This is a schematic diagram of another message processing system provided in an embodiment of this application; Figure 10 This is a schematic diagram of another message processing system provided in an embodiment of this application. Detailed Implementation

[0027] In the embodiments of this application, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different. There is no sequential or hierarchical order among the technical features described by "first" and "second".

[0028] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as exemplary or for example in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Rather, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0029] In the embodiments of this application, at least one can also be described as one or more, and multiple can be two, three, four or more, and this application does not impose any restrictions.

[0030] Furthermore, the network architecture and scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0031] With the rapid development of the domestic and international civil aviation markets and the explosive growth of flight volume, increasingly higher requirements are being placed on the stability, reliability, high throughput, high-speed transmission, proximity access, and accurate synchronization of the real-time civil aviation revenue assessment system and the flight dynamic pricing system.

[0032] In related technologies, traditional synchronization systems face a massive number of message synchronization requests at the same time, requiring second-level synchronization latency, rapid calculation and distribution. If a centralized forwarding synchronization system technology and architecture design are still used, it will lead to congestion of the message synchronization link, client response timeouts, and serious impact on business efficiency.

[0033] Based on this, this application provides a message processing method. By receiving a data change request and immediately sending a cache check delay message to the distributed message cluster, a consistency check trigger point can be pre-set before data writing, avoiding business errors caused by inconsistencies between cache and storage node data due to synchronization delays or anomalies. Secondly, the business change data is written to the local storage node before sending the synchronization message, ensuring the reliability of data persistence and preventing dirty reads caused by messages being delivered before data is written to the database. Finally, the synchronization message includes a data identifier, business change data, and a sending timestamp, enabling the target side to accurately determine the update order based on the timestamp and complete the local cache refresh, thereby achieving eventual consistency between the source and target caches. The entire process ensures the accuracy and fault tolerance of data synchronization while avoiding network overhead caused by frequent cross-node queries, thus significantly improving the overall system throughput and response speed, and ultimately effectively improving business efficiency.

[0034] The solution provided in this application can be applied to Figure 1 In the message processing system shown, such as Figure 1 This is a schematic diagram of the structure of a message processing system provided in an embodiment of this application.

[0035] For example, Figure 1 The illustrated message processing system is deployed across multiple data centers, such as Data Center A, Data Center B, and Data Center C. Each data center operates independently within its internal network, while the data centers are interconnected via high-speed leased lines or Virtual Private Networks (VPNs) to achieve real-time data synchronization across geographical regions.

[0036] Each data center includes: a local storage node, a distributed message cluster, a message synchronization component, and multiple service synchronization devices.

[0037] In some embodiments, local storage nodes are used to store business messages from the data center.

[0038] For example, local storage nodes can be implemented using a distributed database cluster, such as the Apache Cassandra distributed storage cluster. This cluster consists of at least three physical nodes, each storing a copy of the data, with the number of redundant copies controlled by a replication factor parameter.

[0039] Taking data center A as an example, when the business system generates a flight dynamic pricing message (such as the discount coefficient of 0.85 corresponding to flight number "CA1234"), the message will be written to multiple nodes of the distributed storage cluster to ensure that the failure of any single node will not lead to data loss.

[0040] Local storage nodes also support strongly consistent read and write operations, meaning that read operations will wait for multiple replicas to return results to ensure that the data is up-to-date, and write operations will synchronously write to a legally required number of replicas before returning success.

[0041] In some embodiments, a distributed messaging cluster is used to transmit messages between multiple data centers.

[0042] For example, a distributed messaging cluster can be built on Apache RocketMQ and deployed within each data center. The distributed messaging cluster (denoted as RocketMQ) includes name service nodes and broker nodes, where the name service nodes are used for service discovery and routing management, and the broker nodes are used for message storage and forwarding.

[0043] Taking a price change message sent from data center A to data center B as an example, the message is first sent by the RocketMQ producer in data center A to the local broker node. After the broker node persists the message to the local disk, it pushes the message to the RocketMQ consumer in data center B through the cross-data center transmission channel.

[0044] During this process, the distributed message cluster also supports message broadcast mode and cluster mode. Broadcast mode is used to deliver a message to all subscribed consumer instances, while cluster mode is used to distribute messages evenly to multiple consumer instances.

[0045] In some embodiments, the message synchronization component is connected to a local storage node and a distributed message cluster to retrieve messages from the local storage node and send them to other data centers in multiple data centers via the distributed message cluster.

[0046] For example, the message synchronization component can be implemented using the RocketMQ-Replicator middleware. This component runs as an independent process within each data center. One end establishes a persistent connection with the local storage node (such as a Cassandra cluster) through a database connection pool, while the other end connects to the distributed message cluster through the RocketMQ producer client. Taking data center A as an example, when a new business message is added to the local storage node (such as the seat inventory of a flight changing from 50 to 45), RocketMQ-Replicator will detect the data change in real time through Change Data Capture (CDC) technology. Specifically, the component periodically polls the Cassandra write log or uses Cassandra's trigger mechanism to obtain incremental data, then encapsulates the changed data into a standard format message (including data identifier, business change content, and timestamp), and then calls the RocketMQ producer interface to send the message to a preset topic in the distributed message cluster.

[0047] For example, the topic name is "price". This message will then be pushed to the corresponding consumers in data centers B and C respectively through the cross-data center transmission mechanism of the distributed message cluster.

[0048] In some other embodiments, the message synchronization component is also used to receive messages from other data centers via a distributed message cluster and write them to local storage nodes.

[0049] For example, the RocketMQ-Replicator component in data center B subscribes to the topic price through a consumer client. When a message sent by data center A arrives at the distributed message cluster in data center B, the message synchronization component in data center B pulls the message, parses the business data in the message body (such as flight number, changed inventory value), and then calls the write interface of the local storage node to persist the business data to the Cassandra cluster in data center B.

[0050] During the write process, the message synchronization component performs conflict detection and deduplication logic, such as checking whether the unique identifier in the message has already been processed to avoid duplicate writes. Upon successful write, the message completes data synchronization from data center A to data center B. If the write fails (e.g., network jitter causing temporary unavailability of storage nodes), the message synchronization component will continuously attempt retrying according to a preset retry strategy (e.g., exponential backoff algorithm, with retry intervals increasing by 1 second, 2 seconds, 4 seconds, and a maximum of 10 retries) until the write succeeds or the retry limit is exceeded. An exception log is then recorded for maintenance personnel to handle.

[0051] In some embodiments, multiple service synchronization devices communicate with local storage nodes, and each service synchronization device is configured with a local cache unit.

[0052] For example, a service synchronization device can be a server instance with a business system deployed, such as a containerized application running a flight dynamic pricing service. Each container instance embeds a local cache unit. The local cache unit is implemented using the Ehcache framework and deployed in the memory space of the business application process. Taking data center A as an example, three service synchronization devices can be deployed in this data center, denoted as Service 1, Service 2, and Service 3. Each service synchronization device establishes a communication link with a local storage node (such as a Cassandra cluster) through Java Database Connectivity (JDBC) and connects to a distributed message cluster through a RocketMQ client. The local cache unit is used to store hot business data. For example, the cache capacity for frequently queried flight price information can be configured to a maximum of 1000 records, with the expiration strategy using the Least Recently Used (LRU) algorithm. When a service synchronization device (such as Service 1) finds a record in its local cache, it can directly return the cached result in memory without needing to access the underlying Cassandra cluster across the network. This reduces data read latency from milliseconds (approximately 10 to 50 milliseconds for cross-network access) to microseconds (approximately 1 to 5 microseconds for local memory access). When the local cache misses, the device will retrieve the data from its local storage node and write it to the local cache for subsequent requests.

[0053] For example, such as Figure 2 This is a schematic diagram of the structure of a message processing system provided in an embodiment of this application. Figure 2 The message processing system shown includes a presentation layer, an access layer, a service layer, and a data layer.

[0054] In some embodiments, the presentation layer includes a front-end and an interface.

[0055] For example, the front end can be a client application for revenue managers or flight dispatchers.

[0056] For example, a web-based management console running on a personal computer, or a user interface (UI) embedded in a company's internal operating system. This front end is used to display real-time flight revenue assessment results, price trend changes, and data synchronization status.

[0057] Regarding the interface, the presentation layer provides a Representational State Transfer (REST) ​​style Application Programming Interface (API). For example, it receives price change requests from external systems via the Hypertext Transfer Protocol (HTTP) endpoint " / api / price / update". Taking flight "CA1234" as an example, the external dynamic pricing engine calls this interface, passing in the flight number, cabin class (e.g., Economy Class Y), and the adjusted ticket price (e.g., 880 yuan). The presentation layer receives the request and forwards it to the access layer for processing.

[0058] In some embodiments, the access layer includes access layer components.

[0059] For example, access layer components may include a gateway, a load balancer, and an authentication module. The gateway serves as the unified entry point for all external requests, and is responsible for request routing, traffic coloring (labeling requests of different business types), and rate limiting and circuit breaking.

[0060] For example, if the gateway is configured to accept a maximum of 5000 requests per second, and the instantaneous request volume exceeds this threshold, the gateway will reject the excess requests using the token bucket algorithm and return an HTTP status code 429 to prompt the client to retry later. The load balancer employs a weighted round-robin strategy, distributing authenticated and routed requests to different service synchronization device instances in the downstream service layer. Taking data center A as an example, assuming this data center has three service synchronization devices (Service 1, Service 2, Service 3) with processing capacity weights set to 3, 2, and 1 respectively, the load balancer will proportionally distribute requests to the three devices to avoid overloading a single device. The authentication module verifies the caller's identity by checking the JSON Web Token (JWT) carried in the request. The token's validity period is set to 30 minutes, and repeated authentication is not required within this period.

[0061] In some embodiments, the service layer includes multiple data centers (such as data center A, data center B, and data center C) and clients. Each data center includes a service synchronization device, a distributed message cluster, and a message synchronization component.

[0062] For example, the service layer is the core logic processing layer in this application embodiment, and adopts a fully decentralized architecture. Multiple data centers are deployed in different geographical locations, such as data center A in Beijing, data center B in Shanghai, and data center C in Guangzhou. The centers are interconnected via high-speed dedicated lines, and the network round-trip time (RTT) is controlled within 50 milliseconds. Within each data center, the service synchronization device is a stateless business container instance, and each instance is configured with a local cache unit (using the Ehcache framework, with a memory limit of 2 gigabytes and a cache entry limit of 100,000). When a client initiates a data read request to the service synchronization device through the access layer (e.g., querying the current discount coefficient of flight "CA1234"), the device first accesses the local cache: if the cache is hit, the result is returned directly in nanoseconds to microseconds; if the cache is not hit, the data is loaded from the local storage node of the data layer and the cache is populated. The distributed message cluster is implemented using Apache RocketMQ. Each data center independently deploys a RocketMQ cluster, including at least three broker nodes and two name server nodes to ensure high availability. The message synchronization component, RocketMQ-Replicator, runs as a daemon process. It listens to change logs on local storage nodes (e.g., Cassandra's commit log) and encapsulates the changes into messages before sending them to other data centers via a distributed message cluster. For example, if a ticket price change message occurs in data center A: the message is first pushed by RocketMQ-Replicator in data center A to the distributed message clusters in data centers B and C, and then written to the local storage nodes of their respective data centers by their respective message synchronization components, thus achieving eventual consistency across multiple data centers.

[0063] In some embodiments, the data layer includes local storage nodes and local cache units.

[0064] For example, local storage nodes are used to provide persistent storage capabilities, ensuring that business data is not lost after a system power outage or restart. In this embodiment, the local storage nodes are constructed using a distributed database Cassandra cluster. Each data center independently deploys one Cassandra cluster, with a cluster size of at least three physical nodes. Cassandra's data model adopts a wide column structure.

[0065] Figure 3 This is a flowchart illustrating a message processing method provided in an embodiment of this application. The method is applied to a source-side service synchronization device that acts as a message sender.

[0066] like Figure 3 As shown, the message processing method provided in this application embodiment may include: S301: The client sends a data change request to the source-side service synchronization device. Correspondingly, the source-side service synchronization device receives the data change request sent by the client.

[0067] The data change request includes data identifiers and business change data.

[0068] For example, the client can be the pricing engine module in the revenue management system, the data identifier is the unique flight code (e.g., "CA1234|2025-10-01|Y", indicating the Y class of flight CA1234 on October 1, 2025), and the business change data is the adjusted ticket price (e.g., from 880 yuan to 950 yuan) or the discount factor (e.g., from 0.85 to 0.92).

[0069] S302: The source-side service synchronization device sends a cache check delay message to the distributed message cluster based on the data change request.

[0070] In this embodiment of the application, the cache check delay message is used to trigger the consistency check of the local cache unit in the source-side service synchronization device when the time elapsed since its sending reaches the target synchronization delay.

[0071] In some embodiments, the target synchronization delay can be determined by: obtaining a network delay estimate between the source-side service synchronization device and the target-side service synchronization device, and obtaining the current load status index of the source-side service synchronization device, and then obtaining the target synchronization delay based on the network delay estimate and the load status index.

[0072] For example, based on the load status index, the corresponding processing time is determined, and the network latency estimate and processing time are summed to obtain the baseline latency. Then, based on the baseline latency and the preset redundancy coefficient, the target synchronization latency is obtained. The preset redundancy coefficient is used to characterize the system's fault tolerance adjustment capability.

[0073] For example, assuming the measured network round-trip time (RTT) between the source-side service synchronization device (deployed in a Beijing data center) and the target-side service synchronization device (deployed in a Shanghai data center) is 30 milliseconds, and the current load status indicators of the source-side device are CPU utilization of 60% and memory utilization of 50%, the estimated processing time is 20 milliseconds, then the baseline latency is 50 milliseconds. If the preset redundancy coefficient is 1.2 (i.e., allowing 20% ​​additional fault tolerance time), then the target synchronization latency is 60 milliseconds. This latency serves as the delay time for cached check delay messages; the check logic is only activated 60 milliseconds after the message is sent.

[0074] In one scenario, the source-side service synchronization device (referred to as Service 1) receives a price change request from a client at time T0 (e.g., 10:00:00.000). It immediately sends a cache check delay message to the RocketMQ cluster. The message body carries the data identifier "CA1234|2025-10-01|Y" and the target synchronization delay of 60 milliseconds. The message delivery delay is set to the current time plus 60 milliseconds (i.e., 10:00:00.060). Simultaneously, Service 1 continues to execute the subsequent data persistence and synchronization message sending processes.

[0075] S303: The source-side service synchronization device writes the business change data to the corresponding local storage node.

[0076] For example, Service 1 changes the price of the Y class for flight "CA1234" from 880 yuan to 950 yuan, and writes this business data to the corresponding row of the "flight_price" table in the local Cassandra cluster via a JDBC connection. The consistency level is set to QUORUM during the write operation, ensuring that at least two replica nodes confirm the write success before returning. This write operation is completed at time T1 (e.g., 10:00:00.015).

[0077] S304: Upon completion of the write operation, the source-side service synchronization device sends a synchronization message to the distributed message cluster. Correspondingly, the distributed message cluster receives the synchronization message from the source-side service synchronization device.

[0078] The synchronization message includes at least a data identifier, business change data, and a sending timestamp, which is used to instruct the target-side service synchronization device of the message receiver to update the corresponding local cache unit.

[0079] After the data is successfully written to the database, Service 1 immediately sends a synchronization message to the topic "price_sync" in the RocketMQ cluster. The message body includes: data identifier "CA1234|2025-10-01|Y", business change data (new price 950 yuan, old price 880 yuan), operation type "UPDATE", and sending timestamp T1 (a Unix timestamp accurate to milliseconds, such as 1698765600015). RocketMQ broadcasts this message to all service synchronization devices that have subscribed to this topic (including Service 1 itself, as well as Services 2 and 3 within the data center, and even Services 4 and 5 across data centers). Upon receiving the message, each device updates its local cache unit according to the message content.

[0080] Figure 4This is a flowchart illustrating another message processing method provided in an embodiment of this application. The method is applied to a source-side service synchronization device that acts as a message sender.

[0081] like Figure 4 As shown, the message processing method provided in this application embodiment may include: S401: Perform consistency check if the time elapsed since the self-buffered check delayed message is sufficient to meet the target synchronization delay.

[0082] In this embodiment of the application, the cache check delay message sent in step S302 is woken up after the delay time arrives, and the consistency check is performed by the source-side service synchronization device.

[0083] In some embodiments, if the time elapsed since the self-caching check delay message is sent meets the target synchronization delay, the first data corresponding to the local cache unit of the source-side service synchronization device is obtained according to the data identifier, and the second data corresponding to the data identifier is obtained from the local storage node. Then, the first data and the second data are compared for consistency to obtain the comparison verification result.

[0084] The first data is the current cached data of the local cache unit; the second data is used to represent the data source of the standard.

[0085] For example, based on the data identifier "CA1234|2025-10-01|Y", the corresponding first data in the local cache unit of the source-side service synchronization device (such as Service 1) is retrieved. For instance, the local cache unit stores the price of the Y class of this flight as 950 yuan and records the cache update timestamp as T1 (10:00:00.015). This first data is the value currently in the cache.

[0086] Secondly, retrieve the second data corresponding to the same data identifier from the local storage node (such as a Cassandra cluster). The source-side service synchronization device (such as Service 1) initiates a strong consistency read request (with consistency level set to QUORUM) to the local storage node to read the current price of the flight. Assume the read result is 950 yuan, and the last update timestamp of the record is also T1.

[0087] Then, the first data and the second data are compared for consistency to obtain the comparison verification result. The source-side service synchronization device (such as Service 1) compares the two price values ​​and update timestamps: if the two are completely consistent (both are 950 yuan, and the timestamps are both T1), the verification is deemed to have passed, and no repair operation is required; if the first data is 950 yuan but the second data is still 880 yuan due to some anomaly (such as the previous write returning success but some replicas not being synchronized in time), or the first data is missing, the verification result is deemed to be inconsistent.

[0088] S402: If the comparison and verification results are inconsistent, update the local cache unit based on the second data and generate an abnormal synchronization message.

[0089] For example, when the source-side service synchronization device (such as Service 1) detects that the price of 880 yuan in the local cache is inconsistent with the price of 950 yuan in the storage node, it immediately updates the entry corresponding to the data identifier in the local cache to the second data (950 yuan), and records a consistency repair log, which includes the data identifier, the value before repair, the value after repair, and the repair timestamp.

[0090] Furthermore, the source-side service synchronization device (such as Service 1) generates an abnormal synchronization message corresponding to the data identifier and sends it to the distributed message cluster. The message body of this abnormal synchronization message includes: the data identifier "CA1234|2025-10-01|Y", the expected correction value of 950 yuan, the exception type "CACHE_STALE", and the identifier of the source-side service synchronization device. The message topic is set to a cache invalidation topic for all target-side service synchronization devices to subscribe to.

[0091] Among them, the abnormal synchronization message is used to instruct the target-side service synchronization device to set the data corresponding to the data identifier in the corresponding local cache unit to an invalid state, and to send a data recovery request to the source-side service synchronization device according to the data identifier.

[0092] For example, after receiving the abnormal synchronization message, the target-side service synchronization device (such as Service 4) located in the Shanghai data center parses the data identifier "CA1234|2025-10-01|Y" and immediately marks the corresponding entry in its local cache as invalid. Subsequently, Service 4 initiates a data recovery request to the source-side service synchronization device (such as Service 1) based on the data identifier. This request is sent via Remote Procedure Call (RPC). Upon receiving the request, the source-side service synchronization device (such as Service 1) returns the latest data (950 yuan) in the current storage node. The target-side service synchronization device (such as Service 4) then rewrites this data into its local cache, thereby ensuring eventual consistency of the caches across all data centers.

[0093] Through the above steps S401 and S402, this application embodiment not only realizes the conventional synchronous message broadcast update, but also actively discovers and repairs possible omissions or inconsistencies in the local cache through a delay check mechanism. At the same time, when necessary, it links all target-side devices through abnormal synchronous messages to perform cache invalidation and active recovery, forming a closed-loop consistency guarantee system, which effectively avoids the problem of dirty cache reads caused by abnormal scenarios such as network jitter and instantaneous node failure.

[0094] Figure 5 This is a flowchart illustrating another message processing method provided in an embodiment of this application. The method is applied to a source-side service synchronization device that acts as a message sender.

[0095] like Figure 5 As shown, the message processing method provided in this application embodiment may include: S501: The client sends a data read request to the source-side service synchronization device. Correspondingly, the source-side service synchronization device receives the data read request sent by the client.

[0096] The data read request includes the identifier of the data to be read.

[0097] For example, the client sends a data read request to the source-side service synchronization device (such as Service 1), which is passed through the presentation layer's REST interface (such as an HTTP GET request). The data read request includes the identifier of the data to be read, which in this example is "CA1234|2025-10-01|Y", that is, querying the Y class price of flight CA1234 on October 1, 2025.

[0098] S502: The source-side service synchronization device queries the local cache unit based on the identifier of the data to be read to obtain the cache hit status.

[0099] For example, after receiving the request, the source-side service synchronization device (such as Service 1) first checks whether a corresponding cache entry exists in its local cache unit based on the identifier of the data to be read. The local cache unit uses the Ehcache framework, and its internal storage structure is a key-value pair (the key is a data identifier string, and the value is an object containing information such as price and timestamp). The source-side service synchronization device (such as Service 1) uses "CA1234|2025-10-01|Y" as the key for the lookup.

[0100] It should be noted that cache hit status includes both hit and miss.

[0101] S503: When the cache hit status is a miss status, the source-side service synchronization device obtains the business data corresponding to the data identifier to be read from the local storage node and writes the business data into the local cache unit.

[0102] For example, if flight data has never been queried, or if a cached entry has been automatically cleared due to exceeding its expiration time (e.g., 5 minutes), or has been evicted by the Least Recently Used algorithm due to memory capacity limitations, the source-side service synchronization device (e.g., Service 1) will not be able to find the corresponding data in its local cache, and the cache hit status will be "missed". In this case, the source-side service synchronization device (e.g., Service 1) will retrieve the corresponding business data from its local storage node (i.e., the Cassandra cluster) based on the identifier of the data to be read.

[0103] Specifically, the source-side service synchronization device (such as Service 1) generates the query language statement: "SELECT price, update_timestamp FROM flight_price WHERE flight_number='CA1234' AND departure_date='2025-10-01' AND cabin_class='Y'", and sets the consistency level to QUORUM to ensure that the latest data confirmed by a majority of replicas is read. Assume Cassandra returns a price of 950 yuan, and the update timestamp is T1 (10:00:00.015).

[0104] After the source-side service synchronization device (such as Service 1) obtains the business data, it returns the data to the client as an HTTP response and simultaneously writes the data to a local cache unit. During the write operation, the cache key is set to "CA1234|2025-10-01|Y", the cache value is the price of 950 yuan and the corresponding timestamp, and the expiration time of the entry is reset to the current time plus 5 minutes. In this way, subsequent read requests for the same data identifier can directly hit the local cache without needing to penetrate to the persistent storage node again, thus significantly reducing the average read latency and alleviating the query pressure on the underlying storage nodes.

[0105] S504: When the cache hit status is "missed," the source-side service synchronization device sends a response message to the client. Correspondingly, the client receives the response message sent by the source-side service synchronization device.

[0106] For example, assuming that after the aforementioned write request processing steps S301 to S304 are completed, the local cache of the source-side service synchronization device (such as Service 1) has successfully stored the flight price of 950 yuan, and the cache entry has not expired (e.g., the default expiration time can be set to 5 minutes). Then, the cache hit status of this query is "hit". At this time, the source-side service synchronization device (such as Service 1) directly reads the price value of 950 yuan from memory and returns it to the client via an HTTP response. The entire process typically takes between 1 and 5 microseconds, achieving a fast response time of less than milliseconds.

[0107] Figure 6 This is a flowchart illustrating another message processing method provided in an embodiment of this application. The method is applied to a target-side service synchronization device that acts as a message receiver.

[0108] like Figure 6 As shown, the message processing method provided in this application embodiment may include: S601: The distributed message cluster sends a synchronization message to the target-side service synchronization device. Correspondingly, the target-side service synchronization device receives the synchronization message from the distributed message cluster.

[0109] In this embodiment, the synchronization message is sent by the source-side service synchronization device of the message sender after successfully writing the data change request sent by the client into the local storage node.

[0110] Synchronization messages include, but are not limited to, data identifiers, business change data, and sending timestamps.

[0111] For example, suppose the source-side service synchronization device (deployed in the Beijing data center, such as Service 1) processes a flight price change request: adjusting the price of flight "CA1234" on October 1, 2025, in the Y class from 880 yuan to 950 yuan. After the data is successfully written to the Cassandra cluster in the Beijing data center, Service 1 sends a synchronization message to the distributed message cluster. The subject of this message is "price_sync", and the message body includes: data identifier "CA1234|2025-10-01|Y", business change data (new price 950 yuan, old price 880 yuan), operation type "UPDATE", and sending timestamp T_send (e.g., Unix millisecond timestamp 1698765600015, corresponding to Beijing time 10:00:00.015 on October 1, 2025).

[0112] The distributed message cluster broadcasts the message to all consumers subscribed to the topic. Among them, the target-side service synchronization device (such as Service 4) located in the Shanghai data center, as one of the consumers, receives the synchronization message via client-side pull or passive push. Upon receiving the message, the target-side service synchronization device first parses the topic, tags, and other information in the message header to confirm that the message type is a price synchronization message. Then, it extracts the data identifier, business change data, and sending timestamp from the message body, preparing to perform a local cache update.

[0113] S602: The target-side service synchronization device updates the local cache unit based on the data identifier and service change data.

[0114] In some embodiments, the receiving timestamp when the synchronization message is received is obtained, and the current transmission delay is determined based on the receiving timestamp and the sending timestamp. Then, if the current transmission delay does not exceed the target synchronization delay, the service change data is written to the local cache unit.

[0115] The target synchronization delay is the delay estimated by the source-side service synchronization device based on load status indicators and network latency.

[0116] For example, when the target-side service synchronization device receives a synchronization message, it records a local receiving timestamp T. receive (For example, 1698765600045, which is 10:00:00.045). Calculate the current transmission delay ΔT = T receive - T send =45 milliseconds - 15 milliseconds = 30 milliseconds. The source-side service synchronization device pre-sets the target synchronization latency to 60 milliseconds based on its own load status (CPU utilization 60%, processing time estimated at 20 milliseconds) and the network latency estimate between the two data centers (round-trip time RTT approximately 30 milliseconds, one-way approximately 15 milliseconds). Since 30 milliseconds < 60 milliseconds, the transmission latency is determined to be within the normal range, not exceeding the target synchronization latency, and the target-side service synchronization device writes the service change data to its local cache unit. If network congestion causes ΔT to exceed 60 milliseconds (e.g., reaching 80 milliseconds), the source-side latency check and verification mechanism may be triggered, but the target side still performs the write (only recording a latency alarm) to ensure eventual consistency.

[0117] In some embodiments, writing business change data to the local cache unit includes either of the following cases (1) and (2).

[0118] Case (1): If the sending timestamp is later than the second timestamp, the business change data will be determined as the target data content according to the preset writing rules.

[0119] For example, in the local cache unit of the target-side service synchronization device, each cache entry, in addition to storing the business data value, also includes an update timestamp (i.e., the sending timestamp of the last synchronization message written to that entry). This timestamp is denoted as the second timestamp (i.e., T). second When the target-side service synchronization device receives a new synchronization message, it first queries the local cache based on the data identifier "CA1234|2025-10-01|Y" to obtain the second timestamp in the current cache entry.

[0120] Assuming the local cache already contains cached data for this flight, with a price of 880 yuan, and a corresponding second timestamp of 1698765600000 (indicating this data was written by an earlier change message at 10:00:00.000), the new message's sending timestamp T... send = 1698765600015, obviously T send >T second (That is, the sending timestamp is later than the second timestamp), indicating that the new message carries updated data. According to the preset writing rules (i.e., data with updated timestamps has higher priority), the target-side service synchronization device identifies the business change data (i.e., the price of 950 yuan) in the new message as the target data content and uses it to overwrite the old value in the local cache. Simultaneously, it updates the update timestamp of the cached entry to T. send .

[0121] After the write is complete, the price in the local cache becomes 950 yuan, and the timestamp becomes 1698765600015. If a subsequent historical message with an earlier timestamp (e.g., T) is received... send = 1698765599000), according to the same rule, this historical message will be discarded, thus ensuring that the latest version is always retained in the cache.

[0122] Case (2): When the sending timestamp is the same as the second timestamp, the target data content to be written to the local cache unit is determined according to the preset writing rules.

[0123] Among them, the preset writing rules are used to determine the target data content based on the priority of the data source.

[0124] For example, in some extreme scenarios, two different synchronization messages may have the same sending timestamp.

[0125] For example, due to an internal clock inaccuracy issue, the source-side service synchronization device sends two messages for the same data identifier within the same millisecond: one from a business change (e.g., a price adjustment from 880 yuan to 950 yuan), and the other from an anomaly recovery process (e.g., a correction message proactively pushed after the source-side delay check detects an inconsistency). Both messages have the same timestamp of 1698765600015. In this case, the timestamp alone is insufficient to distinguish between the older and younger messages.

[0126] The target-side service synchronization device determines the target data content based on the "data source priority" in the preset write rules. These preset write rules can be configured in advance. For example, business change messages have higher priority than anomaly recovery messages; messages from the primary data center have higher priority than those from the backup data center; or messages with larger sequence numbers (SequenceNumber) have higher priority.

[0127] For example, suppose the message header of a business change message contains the source type field "source". type =NORMAL, the exception recovery message contains "source". type =RECOVERY”, the default write rule stipulates that “NORMAL” has a higher priority than “RECOVERY”. Therefore, when the target-side service synchronization device compares two messages, even though the sending timestamps are the same, it selects the data in the business change message (priced at 950 yuan) as the target data content to write to the local cache according to the priority.

[0128] If two messages have the same priority, a deterministic selection can be made based on the globally unique identifier (UUID) carried in the message (e.g., selecting the one with the larger UUID value), or the original cached value can be kept unchanged and a conflict log can be recorded.

[0129] Through the timestamp comparison and priority rules in S602 above, the target-side service synchronization device can achieve correct updates of the local cache in a distributed, decentralized environment, avoiding data corruption caused by network out-of-order messages, message duplication, or clock precision issues, while ensuring eventual consistency of caches across multiple data centers.

[0130] Figure 7 This is a flowchart illustrating another message processing method provided in an embodiment of this application. The method is applied to a target-side service synchronization device that acts as a message receiver.

[0131] like Figure 7 As shown, the message processing method provided in this application embodiment may include: S701: The distributed message cluster sends an abnormal synchronization message to the target-side service synchronization device. Correspondingly, the target-side service synchronization device receives the abnormal synchronization message from the distributed message cluster.

[0132] For example, in the foregoing Figure 3 and Figure 6 Based on the process shown, when the source-side service synchronization device discovers that the data in the local cache and the persistent storage node are inconsistent through the delayed message verification mechanism (for example, the price of flight "CA1234|2025-10-01|Y" in the local cache is still the old value of 880 yuan, while the price in the storage node has been updated to 950 yuan), in addition to repairing its own cache, the source-side service synchronization device will also generate an abnormal synchronization message and send it to the distributed message cluster.

[0133] The topic of this abnormal synchronization message is set to cache invalid. This message includes, but is not limited to, the data identifier "CA1234|2025-10-01|Y", the expected correction value of 950 yuan, the exception type "CACHE_STALE" (cached outdated) and the source device identifier "Service_1_Beijing".

[0134] After receiving the abnormal synchronization message, the distributed message cluster broadcasts it to all target-side service synchronization devices that have subscribed to the topic.

[0135] Assume that both the target-side service synchronization device (e.g., Service 4) located in the Shanghai data center and the target-side service synchronization device (e.g., Service 7) located in the Guangzhou data center subscribe to the cache invalidation topic. RocketMQ pushes this message in parallel to both target-side service synchronization devices (e.g., Service 4 and Service 7). Upon receiving the abnormal synchronization message, the target-side service synchronization device parses the message body to obtain information such as the data identifier, exception type, and correction value, and prepares to perform subsequent invalidation processing.

[0136] S702: The target-side service synchronization device sets the data corresponding to the data identifier in the local cache unit to an invalid state based on the abnormal synchronization message.

[0137] For example, after receiving an abnormal synchronization message, the target-side service synchronization device first extracts the data identifier "CA1234|2025-10-01|Y" from the message body. Then, it calls the interface of the local cache unit to mark the cache entry corresponding to the data identifier as invalid.

[0138] The specific implementation of marking invalidity can be: setting the cached value to a special value (such as null), or retaining the original value but attaching an "invalid" flag (for example, setting a boolean field "valid=false"), while retaining the original data for traceability.

[0139] In this example, the target-side service synchronization device's local cache originally stored the flight price of 950 yuan (this value may have been updated previously through normal synchronization messages). However, to address the consistency risk indicated by the abnormal synchronization message sent from the source side, the target-side service synchronization device marked this entry as invalid. After being marked as invalid, any subsequent read requests targeting this data identifier will not directly use the cached value, but will instead trigger a source lookup (i.e., retrieving the latest data from the local storage node). This avoids the business layer reading incorrect data due to potential inconsistencies in the cache.

[0140] If the entry corresponding to the data identifier does not exist in the local cache, the invalidation state is considered a no-operation (i.e., no operation is required). The target-side service synchronization device records a processing log containing: operation type "INVALIDATE", data identifier, receiving timestamp, and source-side device identifier, for subsequent auditing by operations and maintenance personnel.

[0141] S703: The target-side service synchronization device responds to the failure state by sending a data recovery request to the source-side service synchronization device based on the data identifier.

[0142] For example, after marking a local cached entry as invalid, the target-side service synchronization device immediately sends a data recovery request to the source-side service synchronization device (i.e., the source-side service synchronization device identified in the abnormal synchronization message, Service_1_Beijing) based on the data identifier "CA1234|2025-10-01|Y". This recovery request is implemented through a remote procedure call (RPC) mechanism, for example, using a remote procedure call framework.

[0143] Furthermore, the target-side service synchronization device constructs an RPC request, the request parameters of which include, but are not limited to, the data identifier “CA1234|2025-10-01|Y”, the requester identifier “Service_4_Shanghai”, and the request timestamp. This RPC request is sent to the source-side service synchronization device (such as Service 1) in the Beijing data center via a high-speed dedicated line between data centers.

[0144] Upon receiving a recovery request, the source-side service synchronization device (e.g., Service 1) queries the latest business data (e.g., price 950 yuan, update timestamp 1698765600015) from its local storage node (e.g., Cassandra cluster) based on the data identifier. This data is then encapsulated as an RPC response and returned to the target-side service synchronization device. Upon receiving the response, the target-side service synchronization device writes the latest data to its local cache (overwriting the previous invalid entries and restoring it to a valid state). It can also return the data to waiting clients (if any) as needed. This entire recovery process achieves passively triggered data repair, ensuring eventual consistency between the target-side cache and the source-side storage node. It also avoids frequent requests to persistent storage from the source due to cache invalidation on the target side, concentrating the repair workload on the source-side service synchronization device.

[0145] Through the abnormal synchronization message processing flow described in S701 to S703 above, the target-side service synchronization device in this embodiment can actively respond to consistency anomalies detected by the source side, proactively invalidate suspicious cache entries, and obtain the latest data through a targeted recovery request. This mechanism forms a closed loop with the delayed message verification on the source side: the source side is responsible for detecting anomalies and initiating notifications, while the target side is responsible for executing invalidation and proactive retrieval, thereby achieving rapid self-healing of cross-datacenter caches under a decentralized architecture, significantly improving the data consistency and availability of the system in complex environments such as high concurrency and network fluctuations.

[0146] Figure 8 This is a flowchart illustrating another message processing method provided in an embodiment of this application. The method is applied to... Figure 1 or Figure 2 The message processing system in the middle.

[0147] like Figure 8 As shown, the message processing method provided in this application embodiment may include: S801: The client sends a data change request to the source-side service synchronization device. Correspondingly, the source-side service synchronization device receives the data change request sent by the client.

[0148] For example, the client initiates a data change request to any service synchronization device in the service cluster. This request includes the identifier of the business data and the specific content to be changed, such as a price update operation for flight "CA1234".

[0149] S802: The source-side service synchronization device sends a cache check delay message to the distributed message cluster.

[0150] For example, after receiving a data change request, the source-side service synchronization device (such as Service 1) first sends a cache check delay message to the distributed message cluster (such as RocketMQ). This message carries the data identifier and the target synchronization delay, and is used to trigger a consistency check of the local cache of this node after the delay time is reached.

[0151] S803: The source-side service synchronizes the device to store data in the database.

[0152] For example, the source-side service synchronization device writes the changed business data to the underlying persistent storage node (such as a Cassandra cluster) to ensure reliable data persistence. Subsequent operations can only proceed after the write is successful.

[0153] S804: The source-side service synchronization device sends a synchronization message to the distributed message cluster.

[0154] For example, after the data is successfully stored in the database, the source-side service synchronization device immediately sends a data synchronization message to the distributed message cluster. This message contains a complete data identifier, business change details, and a sending timestamp, and is delivered via broadcast.

[0155] S805: Distributed message cluster broadcasts synchronization messages to the target-side service synchronization device.

[0156] For example, after receiving a synchronization message, the distributed message cluster pushes it in parallel to all service synchronization devices that have subscribed to the topic in a broadcast mode, such as the target service synchronization device, which includes, but is not limited to, Service 2, Service 3, ..., Service N.

[0157] S806: The target-side service synchronization device will synchronize the synchronization message to the local cache.

[0158] For example, after receiving the broadcast synchronization message, each service synchronization device (Service 1, Service 2, Service 3...Service N) parses the business data and sending timestamp in the message body, and updates its local cache accordingly. For instance, Service 2 and Service 3 update flight prices in their local memory, achieving eventual consistency of the cache across nodes.

[0159] S807: Source-side service synchronization device delay message check.

[0160] For example, the cache check delay message sent in S802 is woken up after a preset target synchronization delay and consumed by Service 1. Service 1 checks the latest value of the corresponding data identifier in the local cache of this node and compares it with the standard data in the persistent storage node. If they match, the process ends; if they do not match, a compensation mechanism is triggered (such as reloading data or sending an abnormal synchronization message) to ensure the correctness of the local cache.

[0161] Through the above steps, this embodiment realizes asynchronous updates and consistency checks of local cache in a decentralized service cluster. Each service node is equal in status and there is no master-slave dependency, effectively avoiding single points of failure and performance bottlenecks caused by centralized caching.

[0162] By way of example, this application also provides another message processing system, which is used to implement the above method embodiments.

[0163] like Figure 9 This is a schematic diagram of another message processing system provided in an embodiment of this application. The message processing system includes a processing module 901 and a communication module 902. The processing module 901 is used to execute... Figure 3 The operations S302 and S303 in the illustrated method are used to execute Figure 4 The operations S401 and S402 in the illustrated method are used to execute Figure 5 The operations S502 and S503 in the illustrated method are used to execute Figure 6 The operation of S602 in the illustrated method is used to execute... Figure 7 The illustrated method includes the operations of S702 and S703, and the methods used for execution. Figure 8 The illustrated method includes operations S803, S806, and S807; the communication module 902 is used to execute... Figure 3 The operations S301 and S304 in the illustrated method are used to execute Figure 5 The operations S501 and S504 in the illustrated method are used to execute Figure 6 The operation of S601 in the illustrated method is used to execute... Figure 7 The illustrated method includes the operation of S701 and its execution. Figure 8 The illustrated method includes the operations of S801, S802, S804, and S805.

[0164] In some embodiments, the message processing system described above includes hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-described functions. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] This application embodiment can divide the message processing system into functional modules according to the above method embodiment. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0166] like Figure 10 As shown, another message processing system provided in this application embodiment may include a processor 1101, a bus 1102, a communication interface 1103, and a memory 1104. The processor 1101, the memory 1104, and the communication interface 1103 communicate with each other via the bus 1102. It should be understood that this application does not limit the number of processors and memories in the network device.

[0167] Bus 1102 can be a PCI bus, an Extended Industry Standard Architecture (EISA) bus, or a UB bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1102 may be represented by a single line, but this does not mean that there is only one bus or a bus of one data mode. The bus 1102 may include a path for transmitting information between various components of the network device (e.g., memory 1104, processor 1101, communication interface 1103).

[0168] Processor 1101 may include any one or more processors such as CPU, graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0169] The memory 1104 may include volatile memory, such as random access memory (RAM). The processor 1101 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0170] The communication interface 1103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between network devices and other devices or communication networks.

[0171] The memory 1104 stores executable program code, and the processor 1101 executes the executable program code to implement the functions of the aforementioned method embodiments. That is, the memory 1104 stores instructions for executing the above methods.

[0172] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the methods provided in the above-described method embodiments.

[0173] In another aspect, a computer program product is provided, which includes a computer program or instructions that, when executed by a processor, implement the methods provided in the above-described method embodiments.

[0174] Through the above description of the implementation methods, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the module can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, modules, and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0175] Since the indicator generation module, computer-readable storage medium, and computer program product in the embodiments of this application can be applied to the above method, the technical effects that can be obtained can also be referred to the above method embodiments. The embodiments of this application will not be repeated here.

[0176] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a network device. Of course, the processor and storage medium can also exist as discrete components in the network device.

[0177] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer programs or instructions. When a computer program or instruction is loaded and executed on a computer, the processes or functions of the embodiments of this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable module. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, a computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A message processing method, characterized in that, A source-side service synchronization device applied to the message sending end; the method includes: Receive a data change request sent by the client; the data change request includes a data identifier and business change data; According to the data change request, a cache check delay message is sent to the distributed message cluster; the cache check delay message is used to trigger the consistency check of the local cache unit in the source-side service synchronization device when the time elapsed since the sending reaches the target synchronization delay. Write the business change data to the local storage node corresponding to the source-side service synchronization device; Upon completion of the write operation, a synchronization message is sent to the distributed message cluster. The synchronization message includes at least the data identifier, service change data, and a sending timestamp, and is used to instruct the target-side service synchronization device of the message receiver to update the corresponding local cache unit.

2. The method according to claim 1, characterized in that, The target synchronization delay is determined in the following way: Obtain the network latency estimate between the source-side service synchronization device and the target-side service synchronization device; Obtain the current load status indicators of the source-side service synchronization device; The target synchronization delay is obtained based on the network latency estimate and the load status index.

3. The method according to claim 2, characterized in that, The step of obtaining the target synchronization delay based on the network latency estimate and the load status index includes: Based on the load status indicators, the corresponding processing time is determined; The network latency estimate and the processing time are summed to obtain the baseline latency; The target synchronization delay is obtained based on the reference delay and the preset redundancy coefficient; the preset redundancy coefficient is used to characterize the system's fault-tolerant adjustment capability.

4. The method according to claim 1, characterized in that, The method further includes: If the time elapsed since the cache check delay message is sent meets the target synchronization delay, the first data corresponding to the local cache unit of the source-side service synchronization device is obtained according to the data identifier; the first data is the current cached data of the local cache unit. Obtain second data corresponding to the data identifier from the local storage node; the second data is used to characterize the standard data source; The first data and the second data are compared for consistency to obtain the comparison and verification results. If the comparison and verification results are inconsistent, the local cache unit is updated based on the second data.

5. The method according to claim 4, characterized in that, If the comparison and verification results are inconsistent, the method further includes: Generate an abnormal synchronization message corresponding to the data identifier; Send an abnormal synchronization message to the distributed message cluster; The abnormal synchronization message is used to instruct the target-side service synchronization device to set the data corresponding to the data identifier in the corresponding local cache unit to an invalid state, and to send a data recovery request to the source-side service synchronization device according to the data identifier.

6. The method according to claim 1, characterized in that, The method further includes: Receive a data read request sent by the client; the data read request includes an identifier of the data to be read. Based on the identifier of the data to be read, query the local cache unit of the source-side service synchronization device to obtain the cache hit status; If the cache hit status is a miss status, the business data corresponding to the data identifier to be read is obtained from the local storage node, and the business data is written to the local cache unit.

7. A message processing method, characterized in that, A target-side service synchronization device applied to a message receiving end; the method includes: Receive synchronization messages from a distributed message cluster; the synchronization messages are sent by the source-side service synchronization device of the message sender after successfully writing the data change request sent by the client into the local storage node; the synchronization messages include at least the data identifier, business change data, and sending timestamp; The local cache unit of the target-side service synchronization device is updated based on the data identifier and business change data.

8. The method according to claim 7, characterized in that, The step of updating the local cache unit corresponding to the target-side service synchronization device based on the data identifier and service change data includes: Obtain the timestamp of the received synchronization message; The current transmission delay is determined based on the received timestamp and the sent timestamp; If the current transmission delay does not exceed the target synchronization delay, the service change data is written to the local cache unit; The target synchronization delay is the delay determined by the source-side service synchronization device based on load status indicators and network latency estimation.

9. The method according to claim 8, characterized in that, The step of writing the business change data into the local cache unit includes: Obtain the second timestamp of the historical data corresponding to the data identifier in the local cache unit; Based on the sending timestamp, the second timestamp, and the preset writing rules, the target data content to be written to the local cache unit is determined; The target data content is written to the local cache unit.

10. The method according to claim 9, characterized in that, The step of determining the target data content to be written to the local cache unit based on the sending timestamp, the second timestamp, and the preset writing rules includes: If the sending timestamp is later than the second timestamp, the business change data is determined as the target data content according to the preset writing rules.

11. The method according to claim 9, characterized in that, The step of determining the target data content to be written to the local cache unit based on the sending timestamp, the second timestamp, and the preset writing rules includes: If the sending timestamp is the same as the second timestamp, the target data content to be written to the local cache unit is determined according to the preset writing rules; The preset writing rules are used to determine the target data content based on the priority of the data source.

12. The method according to claim 7, characterized in that, The method further includes: Receive abnormal synchronization messages from the distributed message cluster; According to the abnormal synchronization message, the data in the local cache unit corresponding to the data identifier is set to an invalid state; In response to the failure state, a data recovery request is sent to the source-side service synchronization device based on the data identifier.

13. A message processing system, characterized in that, The system is deployed across multiple data centers, each data center comprising: Local storage nodes are used to store business messages from the data center; A distributed message cluster is used to transmit messages between the multiple data centers; A message synchronization component, connected to the local storage node and the distributed message cluster, is used to retrieve messages from the local storage node and send them to other data centers among the plurality of data centers via the distributed message cluster; and to receive messages from the other data centers via the distributed message cluster and write them to the local storage node. Multiple service synchronization devices are communicatively connected to the local storage node, and each service synchronization device is configured with a local cache unit; The plurality of service synchronization devices are configured to: when acting as message senders, execute the method of any one of claims 1 to 6; and when acting as message receivers, execute the method of any one of claims 7 to 12.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 12.

Images