A local cache synchronization method and system based on microservices and version number mechanism

Abstract

This invention provides a local cache synchronization method and system based on microservices and version number mechanisms, relating to the field of computer systems. During the introduction of the local cache component, this invention automatically registers and updates microservices, obtains cluster information through the microservice discovery mechanism, and maintains a unified version number through a unified version sequencer. This achieves timely synchronization of local caches across multiple instance nodes through both proactive updates to local caches across multiple instances and passive origin pulls. The unified version sequencer makes decisions based on the submitted version number, informing the instance of the newness or oldness of the local cache data. If it is new, it directly returns the data to the external system; if it is old, it retrieves the data from the origin, updates the local version number, and returns the retrieved data to the external system.

Description

Technical Field

[0001] This invention relates to the field of computer systems, and more specifically, to a local cache synchronization method and system based on microservices and version number mechanisms. Background Technology

[0002] In software systems engineering, caching is a crucial function. In a single-machine, single-process environment, stable JVM memory components like Caffeine can be directly used as the machine's memory for caching. However, in a distributed environment consisting of multiple servers, where multiple processes form a cluster, if the traditional JVM memory caching method is still used, each machine is an independent process, and each process has its own independent JVM memory. Therefore, when configuring the JVM memory on one machine, other machines cannot know about the configuration, preventing the new configuration from being shared across the entire cluster.

[0003] In recent years, with the rapid development of internet technology, especially financial technology (Fintech), more and more technologies (big data, distributed systems, artificial intelligence, etc.) are being applied in the financial field, and financial data is growing exponentially. Taking large commercial banks as an example, they typically have hundreds or thousands of business systems and massive amounts of data from hundreds of millions of users. In this context, higher demands are placed on the data storage capacity, concurrency, and response speed. Currently, in the financial field, caching technology is used when accessing financial database tables using the JVM (Java Virtual Machine). This involves loading the database table records to be accessed into the local JVM memory to improve the speed of data access and reduce database pressure.

[0004] To address the above problems, existing technologies mainly offer the following solutions:

[0005] 1. Through message notification mechanism

[0006] (1) Redis pubsub;

[0007] (2) MQ notifies RocketMQ's broadcast mechanism (used in scenarios with strong consistency);

[0008] shortcoming:

[0009] Complexity: Introducing message queues and consumer mechanisms increases the complexity of the system, requiring additional configuration, monitoring, and maintenance work.

[0010] Data consistency: During message transmission, problems such as message loss, duplicate transmission, or out-of-order delivery may occur, requiring additional processing mechanisms to ensure data consistency.

[0011] System latency: Because the message notification mechanism is asynchronous, there is a certain processing delay. After the data source changes, the cache update may not be immediately reflected after message passing and consumer processing.

[0012] Additional overhead: Maintaining message queue systems and consumer mechanisms requires additional resources and costs, including hardware, network bandwidth, manpower, and system maintenance.

[0013] 2. Polling via scheduled tasks

[0014] (1) Scheduled task to poll the source (weak consistency);

[0015] (2) Polling the version number of the scheduled task;

[0016] shortcoming:

[0017] Not real-time: Scheduled tasks are triggered at set time intervals, so the data in the local cache may not be up-to-date during the time period between two synchronizations.

[0018] Resource consumption: Scheduled tasks require certain system resources, such as CPU and memory. If the task frequency is too high or the data volume is too large, it may have a certain impact on system performance.

[0019] Data consistency issues: During the synchronization process, if abnormal situations occur, such as network interruption or data source failure, some data synchronization may fail, resulting in inconsistency between the local cache and the data source.

[0020] Frequent synchronization may increase system stress: If the scheduled task is set too frequently, it may increase the concurrent access load of the system during synchronization, putting pressure on the data source and system performance.

[0021] 3. Actively call via RPC

[0022] (1) Provide an HTTP interface for active refresh (manual configuration);

[0023] (2) Provide a discovery and registration mechanism with dynamic list updates;

[0024] shortcoming:

[0025] Network latency: RPC involves network communication, which may introduce some latency. This can especially impact performance in poor network environments or under high load.

[0026] System complexity: Introducing an RPC mechanism increases system complexity. It requires configuring and managing related service providers and consumers, as well as handling RPC call requests and responses.

[0027] Development and maintenance costs: Using RPC for proactive calls requires developers to be familiar with frameworks such as Dubbo, and to learn and master the relevant APIs and configurations. Additionally, it necessitates the maintenance and management of remote service registration and discovery.

[0028] Centralized dependency: Using RPC to actively call requires relying on remote services, which means that the availability and stability of the system may be affected by the remote services.

[0029] 4. Synchronization via distributed cluster protocol

[0030] (1) Implemented using jgroup;

[0031] (2) Implemented via gossip;

[0032] shortcoming:

[0033] Data conflict resolution: When multiple nodes modify cached data simultaneously, data conflicts may occur. You need to consider how to resolve data conflicts, such as using mechanisms like version numbers or timestamps to determine the latest data.

[0034] High availability and fault tolerance: Distributed cluster protocols can provide high availability and fault tolerance. However, you need to ensure that the cached state can be properly handled and restored in the event of node failure or network anomalies.

[0035] Performance optimization: Distributed synchronization may introduce latency and overhead. You can consider performance optimization strategies such as asynchronous processing and incremental synchronization. Summary of the Invention

[0036] To address the needs outlined in the background technology, the industry primarily employs three main strategies for synchronization: relying on cache invalidation mechanisms, proactive update mechanisms, and delayed asynchronous update mechanisms. This invention combines the first two of these mechanisms and innovatively combines version numbers with financial gateway microservices to maintain local cache synchronization. During the introduction of the local cache component, the microservice is automatically registered and updated, and cluster information is obtained through the microservice discovery mechanism. A unified version sequencer is used to maintain a unified version number, achieving timely synchronization of the local cache across multiple instance nodes through both proactive updates to local caches across multiple instances and passive origin pulls.

[0037] A local cache synchronization method based on microservices and version number mechanism, the specific steps of which include:

[0038] Step 1: Microservice registration, discovery, and proactive updates;

[0039] The three instance nodes of business A are first registered to the registry center through the microservice framework;

[0040] The registry center synchronizes cluster information to each service instance and then to the microservice cluster information via long-lived connections.

[0041] When the cache of the first instance node of business A is updated, the local JVM memory is updated, and the local microservice cluster information is obtained. Based on the cluster information, instances A-2 and A-3 are called respectively to achieve the purpose of proactive update.

[0042] Step 2: Unify the version sequencer;

[0043] When the back-end management and business systems initiate updates, additions, or deletions of business data, business system A will call the atomic controller provided by the unified version sequencer to perform atomic operations according to the scenario and maintain it in the business data version system table. The version of the newly added data will become 1, the version of the updated data will be incremented by 1, and the version of the deleted data will be reduced to -1. A timeout of 5 minutes will be set.

[0044] When external system B calls business A to query business data, in this scenario, business A obtains the business data version information from the cached standardization component under instance A-1, and sends the current business data version information to the unified version sequencer. The sequencer returns an instruction to business instance A-1 based on whether the versions are the same.

[0045] Step 3: Passive Update;

[0046] When external system B calls business system A to query business data, in the scenario where external system B queries business system A, version inconsistency will trigger a back-to-origin request for passive updates.

[0047] Furthermore: if the version to be sent does not exist, then -2 is sent; if the version to be sent is inconsistent with the cached version, then A-1 is informed that the current unified version sequencer contains the business data version, and A-1 is also informed to call the database to retrieve the source and update the local business data version information and cache information; if the version to be sent is consistent with the cached version, then A-1 is informed that there is no need to retrieve the source, and the current cached information of A-1 is directly returned to the outside.

[0048] Furthermore, the source retrieval operation performs the same concurrent operation as the proactive update.

[0049] Furthermore: the local operation is triggered synchronously with the cluster call.

[0050] Furthermore: A local cache synchronization system based on microservices and version number mechanisms, comprising:

[0051] The microservice registration and update module is used to register with the registry center through the microservice framework and synchronize cluster information to each service instance and the microservice cluster information through long connections. This module also includes a sequential locking module to ensure the timeliness of actively updated business cache data.

[0052] The unified version sequencer module is used to call the atomic controller provided by the unified version sequencer to perform atomic operations according to the scenario and maintain them in the business data version system table. It is also used to obtain business data version information from the cache standardization component and send the current business data version information to the unified version sequencer.

[0053] The passive update module is used to trigger a back-to-origin update when versions are inconsistent.

[0054] Furthermore, the terminal device may include a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions that can be executed by the processor. When the terminal device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the deep learning model training method as described in the foregoing embodiments.

[0055] Further: a storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described above.

[0056] Furthermore: a computer program product comprising a computer program that is executed by a processor using the methods described above.

[0057] The beneficial effects of the present invention are as follows: The present invention has the following advantages:

[0058] Advantage 1: Automatic discovery of unified local cache cluster information. Adopting the in-house financial gateway microservice framework allows for proactive maintenance of cache maintenance interfaces through the microservice discovery mechanism, maintaining them in the service registry center. Clustering is then based on business categories, ensuring that local caches for the same business are within the same cluster. The registry center then synchronizes cluster information to each cluster node via long-lived connections. This way, each local cache cluster node is aware of complete cluster information. This discovery approach is significantly more innovative than manual configuration or writing scheduled tasks to modify the database, and the microservice framework can be reused without introducing additional components.

[0059] Advantage 2: Proactively maintains the local cache by calling the cluster list in case of cache changes. When business requirements necessitate changes to the cache, the cache component can update the local cache of its current instance node while simultaneously using a stable retry mode. Leveraging the microservice framework's capabilities, it uses automatically discovered cluster information to proactively maintain the local cache information by calling the local cache maintenance interfaces of other instance nodes on the machine. This proactive notification does not send business data; only after the notification event occurs does the local cache component proactively determine the version with the unified version sequencer device and then update the cached data in the local cache (JVM) by returning the source business data, thus avoiding data version inconsistencies caused by concurrency.

[0060] Advantage 3: Proactive retries maintain cluster stability. Ensuring idempotency and retries requires a clear response.

[0061] Advantage 4: Maintaining a unified version sequencer for ordering. When an instance node in the cluster receives a local cache maintenance notification, the local cache component determines whether it is an add, update, or delete operation based on the type of the notification received. Then, it queries the unified version sequencer according to the type, and the unified version sequencer makes the decision to retrieve, delete, or update the business data in the local cache.

[0062] Advantage 5: When performing passive queries, the latest version is retrieved, ensuring timely awareness of the latest data. While maintaining the local cache, when an external system calls the local cache component to retrieve cached business data, it first queries the unified version sequencer based on the key of the business cache information to be retrieved. During the query, the version number maintained by the business cache in the local cache is submitted. The unified version sequencer makes a decision based on the submitted version number, informing the instance of the newness or oldness of the local cache data. If it is new, it directly returns the data to the external system; if it is old, it retrieves the data from the origin server, updates the local version number, and returns the retrieved data to the external system. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 The flowchart of microservice registration discovery and proactive updating in the method of the present invention is shown.

[0065] Figure 2 A flowchart of the unified version sequencer in the method of the present invention is shown.

[0066] Figure 3 A schematic diagram of the system of the present invention is shown.

[0067] Figure 4 A schematic diagram of the composition of the terminal device of the present invention is shown. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the present invention are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of the present invention. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the present invention. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0069] Furthermore, the embodiments described herein are merely some, not all, of the embodiments of the invention. The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0070] It should be noted that the term "comprising" will be used in the embodiments of the present invention to indicate the presence of a feature subsequently declared, but does not preclude the addition of other features. It should also be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of the present invention, it should also be noted that the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0071] The following is a detailed description of this case, in conjunction with the relevant accompanying drawings in the instruction manual.

[0072] like Figure 1 As shown in Figure 2, this embodiment describes the method of the present invention, which specifically includes the following steps:

[0073] Step 1: Microservice registration, discovery, and proactive updates

[0074] In this step, the three instance nodes of business A are first registered with the registry center through the microservice framework.

[0075] The registry center synchronizes cluster information to each service instance and then to the microservice cluster information via a long connection.

[0076] When the cache of the first instance node of business A is updated, the local JVM memory is updated, and the local microservice cluster information is obtained. Based on the cluster information, instances A-2 and A-3 are called respectively to achieve the purpose of proactive update.

[0077] Active update consistency design:

[0078] Since proactive updates have two entry points—local operation triggering and cluster-wide synchronous call triggering—multiple entry points can lead to concurrency issues. This invention incorporates a sequential locking module to ensure the timeliness of proactively updated business cache data.

[0079] Step 2: Unified Version Sequencer

[0080] There are two business scenarios in this step where a unified version orderer is used, with the main purpose of improving the timely response awareness of cache updates.

[0081] Scenario 1: The back-end management and business systems initiate updates, additions, and deletions of business data. In this scenario, business system A will call the atomic controller (implemented by Lua script) provided by the unified version sequencer to perform atomic operations according to the scenario, maintain the data in the business data version system table, where the version of a newly added data becomes 1, the version of an updated data is incremented by 1, and the version of a deleted or modified data is decremented by -1, with a timeout of 5 minutes set.

[0082] Scenario 2: External system B calls business A to query business data. In this scenario, business A obtains the business data version information from the cached standardization component under instance A-1, and sends the current business data version information to the unified version sequencer. The sequencer returns an instruction to business instance A-1 based on whether the versions are the same. Instruction details: if the sent version does not exist, -2 is sent; if the sent version is inconsistent with the cached version, A-1 is informed that the current unified version sequencer contains the business data version, and A-1 is instructed to call the database to retrieve the origin and update the local business data version information and cache information; if the sent version is consistent with the cached version, A-1 is informed that there is no need to retrieve the origin, and the current cached information of A-1 is directly returned to the external system.

[0083] Step 3: Passive Update

[0084] Following the scenario in Scenario 2 above, when external system B queries business system A, version inconsistency will trigger a source retrieval, known as a passive update. The source retrieval operation must perform the same concurrent operations as the active update to prevent out-of-order concurrency.

[0085] like Figure 3 As shown, the system of the present invention includes:

[0086] The microservice registration and update module is used to register with the registry center through the microservice framework and synchronize cluster information to each service instance and the microservice cluster information through long connections. This module also includes a sequential locking module to ensure the timeliness of actively updated business cache data.

[0087] The unified version sequencer module is used to call the atomic controller provided by the unified version sequencer to perform atomic operations according to the scenario and maintain them in the business data version system table. It is also used to obtain business data version information from the cache standardization component and send the current business data version information to the unified version sequencer.

[0088] The passive update module is used to trigger a back-to-origin update when versions are inconsistent.

[0089] like Figure 4 As shown, the terminal device 6 may include a processor 601, a storage medium 602, and a bus 603. The storage medium 602 stores machine-readable instructions executable by the processor 601. When the terminal device is running, the processor 601 and the storage medium 602 communicate via the bus 603. The processor 601 executes the machine-readable instructions to perform the steps of the local cache synchronization method based on microservices and version number mechanisms as described in the foregoing embodiments. The specific implementation and technical effects are similar and will not be repeated here.

[0090] For ease of explanation, only one processor is described in the terminal device described above. However, it should be noted that in some embodiments, the terminal device of the present invention may also include multiple processors, and therefore the steps performed by one processor described in the present invention may also be performed jointly by multiple processors or individually.

[0091] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A local cache synchronization method based on microservices and version number mechanism, characterized in that, The specific steps include: Step 1: Microservice registration, discovery, and proactive updates; The three instance nodes of business A are first registered to the registry center through the microservice framework; The registry center synchronizes cluster information to each service instance and then to the microservice cluster information via long-lived connections. When the cache of the first instance node of business A is updated, the local JVM memory is updated, and the local microservice cluster information is obtained. Based on the cluster information, instances A-2 and A-3 are called respectively to achieve the purpose of active update. The active update includes sequential locking to ensure the timing of local operation triggering and cluster call synchronization. The active update only notifies the event and does not transmit business data. Step 2: Unify the version sequencer; When the back-end management and business systems initiate updates, additions, or deletions of business data, business system A will call the atomic controller provided by the unified version sequencer to perform atomic operations according to the scenario and maintain it in the business data version system table. The version of the newly added data will become 1, the version of the updated data will be incremented by 1, and the version of the deleted data will be reduced to -1. A timeout of 5 minutes will be set. When external system B calls business A to query business data, in this scenario, business A obtains the business data version information from the cached standardization component under instance A-1, and sends the current business data version information to the unified version sequencer. The sequencer returns an instruction to business instance A-1 based on whether the versions are the same. Step 3: Passive Update; When external system B calls business system A to query business data, in the scenario where external system B queries business system A, version inconsistency will trigger a back-to-origin request for passive update. The back-to-origin operation performs the same concurrent operation as the active update to prevent out-of-order concurrency.

2. The method according to claim 1, characterized in that, If the version to be sent does not exist, then send -2.

3. The method according to claim 1, characterized in that, The local operation is triggered synchronously with the cluster call.

4. The method according to claim 2, characterized in that, If the uploaded version matches the cached version, A-1 is notified that there is no need to go back to the source; the current cached information of A-1 is directly returned to the external source.

5. The method according to claim 2, characterized in that, If the uploaded version is inconsistent with the cached version, A-1 is informed of the current unified version sequencer and the business data version, and A-1 is also informed to call the database to retrieve the source and update the local business data version information and cache information.

6. A local cache synchronization system based on microservices and version number mechanism, characterized in that, include: The microservice registration and update module is used for microservice registration, discovery, and proactive updates. It includes the following steps: The three instance nodes of business A first register with the registration center through the microservice framework; The registration center synchronizes the cluster information to each service instance and the microservice cluster information via a long connection. When the cache of the first instance node of business A is updated, the local JVM memory is updated, and the local microservice cluster information is obtained. Based on the cluster information, instances A-2 and A-3 are called respectively to achieve the purpose of active update. The active update includes sequential locking to ensure the timing of local operation triggering and cluster call synchronization. The active update only notifies the event and does not transmit business data. The unified version sequencer module is used to call the atomic controller provided by the unified version sequencer to perform atomic operations according to the scenario. Specifically, when the back-end management and business systems initiate updates, additions, or deletions of business data, business system A will call the atomic controller provided by the unified version sequencer to perform atomic operations according to the scenario and maintain them in the business data version system table. The version of the newly added data becomes 1, the version of the updated data is incremented by 1, and the version of the deleted data is reduced to -1. A timeout of 5 minutes is also set. When external system B calls business A to query business data, in this scenario, business A obtains the business data version information from the cached standardization component under instance A-1, and sends the current business data version information to the unified version sequencer. The sequencer returns an instruction to business instance A-1 based on whether the versions are the same. The passive update module is used to trigger a source retrieval when versions are inconsistent. Specifically, when external system B calls business A to query business data, if the versions are inconsistent, a source retrieval will be triggered to perform a passive update. The source retrieval operation performs the same concurrent operations as the active update to prevent out-of-order concurrency.

7. A terminal device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the terminal device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-5.

Images