A key synchronization method and system for an embedded cache database
By adopting a two-layer data storage architecture and a three-level consistency guarantee system in the financial cryptography security service platform, the network latency and fault tolerance issues in high-concurrency scenarios are solved, and the system performance is improved by achieving low latency, high consistency and high availability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG CITY COMMERCIAL BANK COOP ALLIANCE CO LTD
- Filing Date
- 2026-06-24
- Publication Date
- 2026-07-21
AI Technical Summary
Existing financial cryptography security service platforms suffer from network communication latency and Redis single-point performance bottlenecks in high-concurrency scenarios. They also have long data synchronization links and insufficient fault tolerance and automatic recovery capabilities in abnormal scenarios, which affect system performance and consistency.
A two-tier data storage architecture is adopted, consisting of a remote database and a local embedded cache database. The key is read from the local cache first, and a three-level consistency guarantee system and circuit breaker recovery mechanism are constructed to ensure data consistency and system robustness.
Significantly reduces transaction response latency, enhances maximum concurrent processing capacity, ensures data consistency and system availability, and provides automated operation and maintenance capabilities.
Smart Images
Figure CN122437650A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cryptographic security services and data storage technology, and more specifically, to a key synchronization method and system for an embedded cache database. Background Technology
[0002] Currently, financial cryptographic security service platforms commonly face significant performance degradation during upgrades and transformations. To improve service response efficiency, these platforms typically employ architectures that introduce local or remote distributed caching to reduce direct access to remote databases. For example, patent application CN116722971A discloses a method for synchronizing cached data between cryptographic service processes. This method uses Redis as a first-level cache and an in-process cache as a second-level cache. It achieves synchronized updates of cached data across multiple processes through Redis's publish / subscribe mechanism, supplemented by a periodic detection mechanism to ensure data consistency. This solution aims to reduce network I / O overhead between the cryptographic service process and the remote cache, thereby alleviating performance bottlenecks in high-concurrency scenarios.
[0003] However, the aforementioned existing technologies still have the following technical problems: First, its data synchronization architecture still relies on independently running remote caching processes such as Redis. The service process and Redis need to communicate via the TCP protocol. In high-concurrency scenarios, network communication latency and single-point performance bottlenecks of Redis are still difficult to completely eliminate, limiting its ability to support millisecond-level response scenarios such as financial transactions.
[0004] Second, the scheme adopts a two-layer structure of "first-level cache + second-level cache". When the second-level cache is modified, it is necessary to notify other processes through the Redis broadcast mechanism to invalidate the corresponding cache line and reread it from the first-level cache. This process involves multiple cross-process network interactions, and the synchronization link is long, which increases the complexity of ensuring data consistency.
[0005] Third, the solution does not fully consider fault tolerance and automatic recovery mechanisms in abnormal scenarios such as file corruption, disk space exhaustion, or service crashes in the cache layer. In abnormal situations, it still needs to rely on a remote database as a backup, and the availability and robustness of the system need to be improved. Summary of the Invention
[0006] To address the technical problems mentioned above, this invention provides a key synchronization method and system for an embedded cache database.
[0007] This invention provides a key synchronization method for an embedded cache database, comprising: A two-tiered data storage architecture is adopted, consisting of a remote database and a local embedded cache database. The remote database is designated as the single source of fact and is used for persistent storage of key data; the local embedded cache database is used to provide local read and write services for key data for the cryptographic service process.
[0008] During the cryptographic service process startup phase, a full data preheating operation is performed, loading the key data stored in the remote database into the local embedded cache database in one go, so that the service has complete key service capabilities after it is ready.
[0009] When the cryptographic service process is processing a normal transaction and reading the key, it first connects to the local embedded cache database to perform the key reading operation; if the local embedded cache database reading fails, it automatically switches to the remote database to perform the key reading, and uses the key data read from the remote database as the basis for subsequent processing.
[0010] When the cryptographic service process executes a key update transaction, it first persists the updated key data to a remote database. After the remote database successfully stores the data, it then synchronously updates the key data to the local embedded cache database and sends a broadcast notification to other nodes in the cluster so that other nodes can update their local cache.
[0011] Build an independent monitoring process to periodically scan for key data changes in the remote database, pull the changed key data to the local machine, and compare it with the corresponding key data in the local embedded cache database. If they are inconsistent, update the local embedded cache database.
[0012] For transactions within the key change window, both the current version key and the most recent historical version key are stored in the local embedded cache database. When a transaction fails due to a key version mismatch, the historical version key and the key in the remote database are read sequentially for retry verification, and the successfully verified key is written back to the local embedded cache database.
[0013] When the local embedded cache database fails and cannot read the key, the circuit breaker mechanism is automatically triggered, skipping the local cache layer and directly using the remote database as the data source to obtain the key; at the same time, a background recovery thread is started to periodically attempt to reconnect to and test the local embedded cache database. Once the test is successful, the circuit breaker state is automatically released, and the local cache-first read mode is restored.
[0014] This invention provides a key synchronization system for an embedded cache database, comprising: The data storage architecture module is used to configure a two-tier data storage architecture consisting of a remote database and a local embedded cache database. The remote database is identified as the single source of truth and persistently stores key data. The local embedded cache database is used to provide local read and write services for key data for the cryptographic service process.
[0015] The full data preheating module is used to perform a full data preheating operation during the cryptographic service process startup phase, loading the key data stored in the remote database into the local embedded cache database all at once.
[0016] The key reading module is used to prioritize connecting to the local embedded cache database to perform key reading operations when the cryptographic service process is handling normal transactions; and to automatically switch to the remote database to perform key reading when the local embedded cache database read fails.
[0017] The key update module is used to prioritize the persistent storage of the updated key data to a remote database when the cryptographic service process executes a key update transaction. After the key data is successfully stored in the remote database, it will synchronously update the key data to the local embedded cache database and send a broadcast notification to other nodes in the cluster.
[0018] The scheduled scanning module is used to build an independent monitoring process, which periodically scans for key data changes in the remote database, pulls the changed key data to the local machine, compares it with the corresponding key data in the local embedded cache database, and updates the local embedded cache database when there is a discrepancy.
[0019] The window period transaction compensation module is used to simultaneously retain the current version key and the most recent historical version key in the local embedded cache database. When a transaction fails due to key version mismatch, it reads the historical version key and the key in the remote database in turn for retry verification, and writes the successfully verified key back to the local embedded cache database.
[0020] The circuit breaker recovery module is used to automatically trigger the circuit breaker mechanism when the local embedded cache database fails and cannot read the key. It directly uses the remote database as the data source to obtain the key and starts a background recovery thread to periodically attempt to reconnect to and test the local embedded cache database. Once the test is successful, the circuit breaker state is automatically released, and the local cache priority reading mode is restored.
[0021] Compared with the closest existing technology, the key synchronization method and system for embedded cache database provided by this invention achieves the following beneficial effects through the organic synergy and overall integration of multiple technical features: I. Eliminate network dependence and significantly reduce transaction response latency In the prior art, Redis is used as the first-level cache, and cross-process network communication between the service process and Redis needs to be carried out through the TCP protocol. In high-concurrency scenarios, network I / O overhead becomes a performance bottleneck. The present invention uses the local embedded cache database Sqlite to replace the remote distributed cache, and the service process directly interacts with the local cache through the native C language API, completely eliminating the network communication link during the cache access process. This technical means is coordinated with the full-volume preheating technical feature. During the startup phase, the remote database key data is loaded into the local cache一次性, ensuring that all key reads during service operation are completed locally without any cross-process network interaction. As a result, the average transaction response time of the system is reduced from 3.54 ms to 1.08 ms, a decrease of 69.5%, meeting the stringent requirements of the financial transaction scenario for millisecond-level responses.
[0022] II. Construct a three-level consistency guarantee system to achieve the balance between performance and consistency The prior art relies on the publish / subscribe broadcast mechanism of Redis to achieve multi-process cache synchronization. This method has problems such as long synchronization links, the risk of broadcast message loss, and a relatively long data inconsistency window period. The present invention constructs a three-level consistency guarantee system of "broadcast real-time synchronization + timed scan correction + abnormal transaction compensation", and a synergistic and complementary relationship is formed between the technical features of each level: The first level, when the key is updated, the "remote first update + UDP broadcast retransmission" mechanism is adopted to ensure that the changed data is propagated to all nodes in the cluster within hundreds of milliseconds, taking into account both the synchronization efficiency and the low-latency requirements; The second level, an independent monitoring process scans the key changes in the remote database within the last 10 seconds at a 5-second interval as a fallback compensation for the broadcast mechanism, effectively solving the problem that some nodes do not receive the broadcast notification due to network packet loss; The third level, for transactions within the key change window period, both the current version and the historical version of the key are retained in the local cache. When specific error codes such as MAC verification failure are detected, an incremental retry verification of "current key → historical key → remote database" is automatically executed, and the key that passes the verification is written back to the local cache. The three-level mechanism cooperates with each other and progresses layer by layer, not only ensuring the ultimate consistency of the key data among the nodes in the cluster, but also locking the read operation to be completed in the local cache, achieving the optimal balance between data consistency guarantee and extreme reading performance.
[0023] III. Establish an abnormal self-healing ability to improve the system availability and robustness Existing technologies do not adequately consider fault tolerance and automatic recovery mechanisms in the event of cache layer anomalies. If Redis service fails or network outages occur, the entire cryptographic service platform becomes unavailable. This invention integrates a circuit breaker mechanism with a window period compensation mechanism, forming a complete self-healing system: when the local embedded cache database experiences file corruption, disk space exhaustion, or abnormal deletion of database files, the system automatically triggers a circuit breaker, skipping the local cache layer and directly using the remote database as the data source to continue providing services, ensuring uninterrupted transaction links. Simultaneously, the background recovery thread attempts to reconnect to and test the local cache database every 10 seconds. Once the fault is resolved, the circuit breaker is automatically lifted, restoring the normal working mode with local cache priority. This circuit breaker recovery mechanism works in conjunction with the aforementioned three-level consistency guarantee mechanism. During the circuit breaker period, data reading still relies on the remote database, without violating the "single source of fact" principle. After the circuit breaker is lifted, the local cache can automatically synchronize data through a timed scanning mechanism without manual intervention. Therefore, the system achieves a seamless transition in cache layer failure scenarios, significantly improving the robustness and automation level of the financial cryptographic service platform under high availability requirements.
[0024] IV. Overall Technical Effects Brought About by Synergistic Technical Features The aforementioned technical features do not function in isolation, but rather work together organically to form a holistic technical solution encompassing "low-latency access, strong consistency assurance, and high availability and fault tolerance": the use of local embedded caching provides the physical basis for eliminating network latency; the three-level consistency assurance system provides logical guarantees for the correctness of the local cache; and the circuit breaker recovery mechanism provides a safety net for cache anomaly scenarios. These three elements are interdependent and complementary, jointly solving the technical challenges of severe performance degradation, complex consistency assurance, and weak anomaly recovery capabilities in existing technologies within a cluster environment. This increases the system's maximum concurrent processing capacity from 20,500 transactions / second to 28,500 transactions / second, an increase of 39%, while simultaneously achieving automated self-healing capabilities with zero human intervention in operations and maintenance. Attached Figure Description
[0025] Figure 1 : Overall flowchart of the key synchronization method for embedded cache database in Embodiment 1 of the present invention; Figure 2 : Flowchart of key reading branches in steps 3 to 4 of Embodiment 1 of the present invention; Figure 3 The flowchart of key update and UDP broadcast synchronization in step 5 of embodiment 1 of the present invention; Figure 4 : Flowchart of the timed scanning incremental synchronization in step 6 of embodiment 1 of the present invention; Figure 5The flowchart of the key change window period transaction anomaly compensation process in step 7 of embodiment 1 of the present invention; Figure 6 Flowchart of cache failure circuit breaking and automatic recovery in step 8 of Embodiment 1 of the present invention; Figure 7 The key synchronization system architecture diagram of the embedded cache database in Embodiment 2 of the present invention. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are for illustrative purposes only and are not intended to limit the invention.
[0027] In this invention, unless otherwise expressly stated, the following terms have the following meanings: Embedded cache database: refers to a lightweight database that does not require a separate service process and runs as a library file embedded in the application process. It adopts a single-node file storage mode, and the application can directly access the database file through the native API without network topology dependency.
[0028] Single source of fact: In a two-tier data storage architecture, the remote database is established as the authoritative source and final basis for key data, while the data in the local embedded cache database is regarded as a copy of the remote database. When the two are inconsistent, the remote database shall prevail.
[0029] Key change window: This refers to the time interval from when the key data in the remote database is updated until the local embedded cache databases of all nodes in the cluster have completed the key synchronization update. During this window, the local caches of different nodes may be in an inconsistent state.
[0030] Circuit breaker mechanism: When a local embedded cache database fails and cannot read the key normally, the system automatically switches the data source, skips the local cache layer, and directly uses the remote database as the source of key acquisition. This is a fault isolation and degradation processing mechanism.
[0031] UDP broadcast retransmission mechanism: When using the User Datagram Protocol (UDP) for key synchronization notification between nodes, the critical synchronization message is sent at least twice with a preset interval (e.g., 0.5 seconds) to reduce the risk of notification failure due to network packet loss.
[0032] Example 1: Key Synchronization Method for Embedded Cache Databases This embodiment describes in detail a key synchronization method for an embedded cache database. The method is applied to a financial cryptographic security service platform, which is deployed in a cluster environment consisting of multiple service nodes, each of which is configured with a local embedded cache database.
[0033] Figure 1 The overall flow of the method is illustrated. The method described in this embodiment can be divided into three progressive levels according to execution sequence: initialization phase, normal operation phase, and exception handling phase. The initialization phase includes steps 1 and 2, used to establish a two-layer storage architecture and complete cache preheating; the normal operation phase includes steps 3 and 5, handling the two core transaction scenarios of key reading and key updating respectively; the exception handling phase includes steps 4, 6, 7, and 8, forming a three-level consistency guarantee system and fault tolerance mechanism. There are clear logical dependencies between the steps: steps 3 and 4 are branch paths of each other; the broadcast synchronization of step 5 and the timed scanning of step 6 constitute complementary guarantees of real-time and timed operations; step 7 serves as a compensation mechanism for step 5 within the window period; and step 8 serves as a fallback for steps 3 and 4 in the event of cache failure.
[0034] Step 1: Build a two-tier data storage architecture A two-tiered data storage architecture is employed, consisting of a remote database and a local embedded cache database. The remote database is designated as the "single source of truth" for the entire system, used to authoritatively and persistently store all key data. A local embedded cache database is configured for each cryptographic service process; in this embodiment, SQLite is chosen as the embedded cache database.
[0035] SQLite is developed in C, making it naturally compatible with financial cryptography service platforms that are also developed in C. It eliminates the need for a separate service process and removes the network topology dependencies required for distributed caching. SQLite uses mmap memory mapping to map database files to the process's address space, transforming file I / O operations into memory read / write operations and avoiding the overhead of traditional system calls. Combined with a write-ahead logging (WAL) model, changes are first written to a separate log file and then merged into the main database upon commit. This model ensures that incomplete transactions can be recovered from system crashes through the log, thus providing the ACID properties of atomicity, consistency, isolation, and durability while meeting the millisecond-level response requirements of financial cryptography services for cached data. SQLite uses a B+ tree as its core index structure, with a default page cache size of 64MB. In high-concurrency read scenarios for financial cryptography services, the page cache can be expanded to 256MB to further improve the hit rate.
[0036] Step 2: Perform full data warm-up. During the cryptographic service startup phase, a separate initialization application is designed to automatically perform a full data preheating operation. When the platform starts, a local embedded cache instance is automatically initialized. After the platform's shared memory module is loaded, the key data is fully loaded from the remote database into the local embedded cache. The embedded cache data is persistently stored as files, with the default storage path being the platform's runtime directory.
[0037] If the connection to the remote database fails, the system immediately releases the occupied shared memory resources and terminates the platform service process, returning an error message. After the embedded cache instance is created, the transaction processing program starts and automatically establishes a connection with the cache database. After the platform service completes all initialization configurations, it enters the ready state, enabling it to immediately possess full key service capabilities.
[0038] After step 2 is completed, the system enters a ready state and can receive and process transaction requests. Subsequent steps 3 through 10 are all executed based on this.
[0039] Figure 2 The detailed branching process of the normal path and abnormal path in the key reading scenario is further illustrated, corresponding to steps 3 and 4 below.
[0040] Step 3: Perform key standardization reading (normal path) When the system is in a ready state, this step is executed upon receiving a transaction request. This step represents the normal path for key reading and branches off from the abnormal path in step 4.
[0041] When processing normal transactions, the platform prioritizes connecting to the local embedded cache database to perform key reading operations. The key reading strategy follows the priority order of "Chinese cryptographic algorithms first, then international algorithms." The platform interacts directly with the cache database through the native SQLite C language API (including sqlite3_open_v2, sqlite3_prepare_v2, sqlite3_step, etc.), avoiding the additional serialization and network overhead of intermediate driver layers such as ODBC or JDBC, and ensuring the shortest data reading path.
[0042] After obtaining the matching key value, the platform assembles it into a transaction message and sends it to the financial data cryptographic device for cryptographic computation. The platform then verifies the transaction result returned by the financial data cryptographic device to determine if the transaction was executed correctly. Once the transaction is complete, the system returns to idle listening mode, awaiting the next transaction.
[0043] Step 4: Perform a key downgrade read (abnormal branch) If an anomaly is detected in the local embedded cache database during step 3, the system automatically switches to this step. This step serves as an exception branch path for step 3, ensuring that transactions can continue to be processed even in the event of a cache failure.
[0044] Specific abnormal scenarios include: missing key, cache library connection failure, abnormal deletion of cache files, disk space exhaustion leading to write failure, and database file corruption. When any of the above abnormalities occur, the system automatically switches to a remote database connection to re-execute key reading, and the reading order also follows the principle of "first the national cryptographic algorithm, then the international algorithm".
[0045] After obtaining a valid key value, it is assembled into a transaction message and sent to the financial data cryptographic device to perform cryptographic operations. The returned result is verified, and the device returns to the idle listening state after the transaction is completed.
[0046] Step 5: Perform key update and synchronization distribution Figure 3 The detailed processing flow for the key update scenario in step 5 is shown, including priority update of the remote database, local cache synchronization, and UDP broadcast notification mechanism. This step is parallel to steps 3 and 4, handling read-type transactions and update-type transactions respectively. After step 5 is completed, the timed scanning baseline update in step 6 and the window period compensation preparation in step 7 will be triggered.
[0047] When the cryptographic service process executes a key update transaction, it prioritizes persistently storing the updated key data in a remote database. Specifically, after the transaction message is sent, it reads the current key-related information to construct a generation context, assembles and generates a key transaction message based on the obtained key information, and sends it to the financial data cryptographic device to execute key generation. If key generation is successful, the generated key is prioritized for persistent storage in a remote database; if remote storage fails, the transaction fails.
[0048] If remote storage is successful, the key value is immediately synchronized and updated to the local embedded cache to ensure efficient retrieval of subsequent transaction key data for this service node. Subsequently, based on the other platform service node information configured in the management console function menu, the list of target node IPs and ports that need to receive updates is parsed and obtained. The updated key value is then pushed to the other platform service nodes in sequence, and each target node stores the key in its local embedded cache after receiving it.
[0049] Key synchronization with other nodes is accomplished via the UDP protocol to ensure low latency on the main transaction link. The reason for using UDP instead of TCP for broadcast notifications is that UDP is a connectionless protocol, eliminating the need for the three-way handshake to establish a connection and the four-way handshake to release it, resulting in lower communication overhead. UDP broadcasts can simultaneously send data packets to all nodes within the same network segment, with the number of network transmissions independent of the number of nodes. In key synchronization scenarios, a small amount of notification packet loss is permissible (compensated by a timed scanning mechanism). Therefore, UDP's high efficiency is more advantageous than TCP's reliability guarantees.
[0050] The platform constructs a dedicated receiving service to receive key synchronization transaction requests from other nodes. This service employs a multi-process architecture to support high-concurrency transaction pushes, with the total number of service processes configured as (number of platform nodes - 1) × 3 to ensure concurrent connection capabilities between peer nodes. To prevent UDP packet loss due to network fluctuations, the platform implements a retransmission mechanism for pushed transactions, sending critical synchronization messages at least twice after a 0.5-second interval to improve the success rate of synchronization transactions.
[0051] Step 6: Perform timed incremental synchronization scan like Figure 4 As shown, this step runs continuously as an independent monitoring process, complementing the broadcast synchronization mechanism in step 5. Step 5's broadcast synchronization handles real-time notifications, while this step handles periodic fallback scans; together, they ensure the eventual consistency of the cluster data.
[0052] Build an independent automated synchronization service to incrementally synchronize key data from a remote database to a local embedded cache according to a preset strategy. After the service program starts, it reads the scan interval parameter. If the parameter value exceeds a reasonable range or is abnormal, a default of 5 seconds is used as the scan interval and thread sleep period.
[0053] Upon initial program execution, the current system time is obtained as the initial reference point. A full key loading operation is performed, and this time point is parsed into a specific date and time format for subsequent comparison. After the full key loading is complete, the currently synchronized timestamp is updated and persisted, and then the program enters a sleep state until the interval period ends.
[0054] Upon wake-up, the system uses the recorded time point as the query condition to retrieve data changes in the remote keystore after that time point. The scanning window is set to the updated key data in the remote database within the last 10 seconds. Setting the scanning window to 10 seconds coordinates with the 0.5-second broadcast retransmission interval and the 5-second timed scan interval. Broadcast notifications are sent twice within 0.5 seconds, theoretically reaching all nodes within 1 second. Setting a 10-second window covers the greatest common divisor of the normal arrival time of broadcast notifications (approximately 1 second) and the scanning cycle (5 seconds), ensuring that even if a node misses multiple broadcast notifications consecutively, it can complete data correction within 10 seconds through timed scanning, while avoiding performance impact due to excessive scan data volume caused by an overly large window.
[0055] If data changes are detected within the specified time period, the latest changed data is fetched and updated to the local embedded cache to ensure that the cached data is up-to-date. If no data changes are detected, the synchronization step is skipped, and the current cache state remains unchanged.
[0056] Update the current time to the new query baseline, record the latest synchronization timestamp, and then enter sleep mode again to wait for the next synchronization cycle. Repeat the above detection, synchronization, and sleep process to achieve long-term, automated incremental synchronization of key data.
[0057] Step 7: Perform transaction anomaly compensation during the key change window period like Figure 5 As shown, this step serves as a supplementary mechanism to step 5, specifically addressing the scenario where the broadcast notification in step 5 fails to reach some nodes in a timely manner due to network packet loss. This step is triggered when a specific error code is detected in step 3 or step 4.
[0058] For special transaction scenarios such as message MAC verification and PIN conversion, the current version key (version N) and the most recent historical version key (version N-1) are simultaneously stored in the local embedded cache database to deal with the key version mismatch problem that may occur during the key change window period—that is, the time gap between when the remote database has been updated but the broadcast notification has not yet reached this node.
[0059] The design logic of the three-step progressive verification is as follows: The first verification uses the local current version key to handle transactions under normal circumstances; if it fails and the error code indicates a MAC verification error or PIN conversion error, it indicates that there may be a key change window period. Therefore, the second verification uses the locally retained historical version key. This scheme can solve the problem that the local node has not yet received the broadcast notification during the window period; if it still fails, it indicates that there may be a large discrepancy between the local node's cache and the remote database. Therefore, the third verification directly reads the remote database and performs the final verification based on the "single source of fact". The three verifications are progressive, covering three scenarios: normal situation, window period delay, and node cache anomaly. This ensures consistency while avoiding unnecessary remote database access overhead.
[0060] The specific processing flow is as follows: When a transaction message is sent, the latest valid key is read first. The transaction message is reassembled based on the obtained key and sent to the financial data cryptographic device for cryptographic calculation. A preliminary judgment is made based on the returned result. If a transaction failure is detected, and the reason for the failure is not a message MAC verification error and the transaction is not a message MAC verification and PIN conversion transaction, a transaction failure result is returned directly.
[0061] If the transaction failure type is "Message MAC Verification" or "PIN Conversion" and the initial verification fails, a second transaction verification is performed using the historical key retrieved from the local embedded cache. If the second verification using the historical key data also fails, the transaction waits for 100ms before switching to the remote database to retrieve the corresponding historical key for a third transaction verification. If the third verification succeeds, a transaction success is returned, and the successfully verified key is written back to the local embedded cache to eliminate state inconsistencies; otherwise, a transaction failure is returned.
[0062] Step 8: Perform cache failure circuit breaking and automatic recovery like Figure 6 As shown, this step serves as a global fallback mechanism for steps 3, 4, and 5. It triggers a circuit breaker when the cache library is detected to be unavailable in step 3 or 4. This step is not constrained by the execution order of other steps and can intervene at any time.
[0063] When a problem occurs in the local embedded cache database that prevents the key from being read, the system automatically triggers a circuit breaker mechanism, skipping the local cache layer and directly using the remote database as the centralized data source to obtain the key, ensuring that the transaction proceeds normally.
[0064] During circuit breaker operation, all transaction processing directly connects to the remote database. Simultaneously, a background recovery thread initiates, attempting to reopen the SQLite cache database and perform a test read / write operation every 10 seconds. If the test operation succeeds, the circuit breaker is automatically lifted, and the data source is switched back to the normal mode with local cache priority. This circuit breaker and automatic recovery mechanism ensures a smooth transition without manual intervention in the event of a cache layer failure.
[0065] Furthermore, this method can add a "Publish Key" function entry to the management platform, supporting key generation and publication. After receiving and completing the key information entry, the management platform automatically pushes the key data to a remote database for persistent storage, ensuring the security and reliability of the source data. In addition, a "Load Key" interaction entry is added. After each cluster platform node receives the load instruction, it triggers an incremental synchronization process, only pulling incrementally changed key data to the local embedded cache to improve system performance and accelerate key activation.
[0066] The platform node IPs deployed using this method are stored in a remote database for management and can be dynamically loaded into shared memory. This allows the broadcast process to dynamically obtain the platform node information that needs to be synchronized, thereby enabling dynamic scaling of the platform cluster.
[0067] Example 2: Key Synchronization System for Embedded Caching Databases This embodiment corresponds to the method described in Embodiment 1, and provides a key synchronization system for an embedded cache database.
[0068] like Figure 7 As shown, the system comprises four layers: an external access layer, a cryptographic service node layer, a data layer, and a management and control layer. The cryptographic service node layer consists of multiple peer service nodes; the diagram exemplifies two nodes, A and B, but can be expanded to N nodes in actual deployment. Each node internally includes a full-scale preheating module, a key reading module, a key update module, a timed scanning module, a window period transaction compensation module, a circuit breaker recovery module, and a local embedded cache database. The deployment of each module corresponds to the phased division of the method steps in Example 1: the full-scale preheating module corresponds to the initialization phase; the key reading and key update modules correspond to the normal operation phase, with the key reading module integrating the branch logic of steps 3 (standardized reading) and 4 (abnormal degradation reading); the timed scanning, window period transaction compensation, and circuit breaker recovery modules correspond to the exception handling phase. These three modules work together to form a three-level consistency guarantee and fault tolerance system.
[0069] The data storage architecture module configures a two-tiered data storage architecture consisting of a remote database and a local embedded cache database. The remote database is designated as the single source of truth and persistently stores key data. The local embedded cache database provides local read / write services for key data to the cryptographic service process. In this embodiment, SQLite is used as the local embedded cache database, employing a single-node file storage mode. It interacts directly with the cryptographic service process via native C language APIs, eliminating network topology dependencies.
[0070] The full data preheating module performs a full data preheating operation during the cryptographic service process startup phase, loading the key data stored in the remote database into the local embedded cache database all at once. This module automatically initializes the local embedded cache instance during platform startup, and after the shared memory module has finished loading, it loads the key data from the remote database in full. If the connection to the remote database fails, it immediately releases the occupied resources and terminates the service process.
[0071] The key reading module is used to prioritize connecting to the local embedded cache database to perform key reading operations during normal transactions in the cryptographic service process. The key reading strategy follows the priority order of "Chinese cryptographic algorithms first, then international cryptographic algorithms". This module is also used to automatically switch to the remote database to perform key reading when there is an error in reading from the local embedded cache database, and to use the key data read from the remote database for subsequent processing.
[0072] The key update module prioritizes persisting the updated key data to a remote database during key update transactions executed by the cryptographic service process. After successful storage in the remote database, it synchronously updates the key data to the local embedded cache database and simultaneously broadcasts a notification to other nodes in the cluster. This module achieves key synchronization between nodes via the UDP protocol, employs a dedicated multi-process architecture for receiving transactions to support high-concurrency transaction pushes, and sends critical synchronization messages at least twice to prevent network packet loss.
[0073] The scheduled scanning module is used to build an independent monitoring process that periodically scans for key data changes in the remote database. It retrieves the changed key data to the local machine and compares it with the corresponding key data in the local embedded cache database. If there is a discrepancy, it updates the local embedded cache database. This module uses a default scan interval of 5 seconds, and the scan window is set to the updated key data in the remote database within the last 10 seconds.
[0074] The window period transaction compensation module simultaneously stores the current version key and the most recent historical version key in the local embedded cache database. When a transaction fails due to a key version mismatch, it sequentially reads the historical version key and the key from the remote database for retry verification, and writes the successfully verified key back to the local embedded cache database. This module is effective for special transaction scenarios such as message MAC verification and PIN conversion.
[0075] The circuit breaker recovery module automatically triggers a circuit breaker mechanism when a failure occurs in the local embedded cache database, making it impossible to read the key. It then directly uses the remote database as the data source to obtain the key and starts a background recovery thread to periodically attempt to reconnect to and test the local embedded cache database. Once the test is successful, the circuit breaker is automatically lifted, and the local cache-first read mode is restored. The default test period for the background recovery thread is 10 seconds.
[0076] The above modules are deployed in a cluster environment consisting of multiple service nodes. The platform IPs of each node are stored in a remote database for management and can be dynamically loaded into shared memory, supporting dynamic expansion of the cluster.
[0077] Example 3: Case Study on the Reconstruction of Financial Cryptographic Security Service Platform Architecture This embodiment applies the technical solution of the present invention to the architecture reconstruction project of a cryptographic security service platform of a financial institution. The institution's original system used ODBC direct connection to a remote database for key data acquisition. After the transformation, it experienced a severe performance degradation, with an average transaction response time of 3.54ms and a maximum concurrent processing capacity of only 20,500 transactions / second, which could not meet the full-stack security support requirements of more than 40 upstream business systems such as mobile banking and corporate online banking.
[0078] After implementing the technical solution of this invention, the system adopts a local embedded caching mechanism to replace the original ODBC direct connection to the remote database mode. During the architecture reconstruction, the platform IP information of each service node is uniformly stored in the remote database for management, and a dynamic loading mechanism is configured. When the platform starts, it automatically performs full data preheating, loading the key data from the remote database into the SQLite local cache of each node. Eight service nodes are deployed in the production environment, each node is configured with a default page cache of 64MB, and the page cache is expanded to 256MB to further improve the hit rate based on the high concurrency characteristics of the business.
[0079] After system launch, it entered the continuous operation monitoring phase. Monitoring data shows that the overall system operation is stable, with no abnormal errors or performance fluctuations. Regarding CPU load, the CPU utilization of application service nodes remains stable, with daily operating indicators all below 10%. Regarding memory resources, the daily peak memory utilization of application service nodes is consistently controlled within 30%, meeting normal operational standards. Transaction response time decreased from an average of 3.54ms to 1.08ms, a reduction of 69.5%; maximum concurrent business processing capacity increased from 20,500 transactions / second to 28,500 transactions / second, an increase of 39%.
[0080] Example 4: Case Study on Anomaly Handling During Key Change Window This embodiment illustrates the practical application effect of the key change window period anomaly compensation mechanism in this invention using a specific transaction scenario.
[0081] During a certain trading session, operations personnel released a new version of the payment key (version N+1) through the management platform to replace the original key (version N). The remote database completed the key update at time T0. Node A, which was responsible for processing check-in transactions, completed its local cache update at time T0+10ms; Node B, due to packet loss during broadcast message transmission, failed to receive the key change notification immediately, and its local cache still retained the version N key.
[0082] At time T0+50ms, a cardholder PIN conversion transaction is assigned to node B for processing. Node B assembles the transaction message using its locally cached version N key and sends it to the financial data cryptographic machine. The cryptographic machine returns a MAC verification failure error code.
[0083] Upon detecting the error code, Node B triggers the window period transaction anomaly compensation process: First, it reads the historical key (i.e., the earlier version N-1 key) stored in its local cache for secondary verification, which also fails. Then, after a 100ms wait, Node B switches to the remote database to retrieve the corresponding key for a third verification. At this point, the remote database stores the version N+1 key, and the third verification succeeds. Node B writes the successfully verified version N+1 key back to its local embedded cache, eliminating the state inconsistency. Subsequent transactions directly use the version N+1 key for processing, and the transaction ultimately returns success.
[0084] This case demonstrates that, in extreme cases where broadcast notifications fail to be delivered in a timely manner due to network packet loss, the anomaly compensation mechanism in the three-level consistency guarantee system of this invention can effectively provide a safety net, ensuring that transactions do not fail due to cache inconsistency.
[0085] Example 5: Case Study of Cache Failure Circuit Breaker and Automatic Recovery This embodiment illustrates the practical application effect of the circuit breaking and automatic recovery mechanism in this invention using a specific fault scenario.
[0086] During peak trading hours on a weekday, the disk space of a certain password service node was exhausted due to abnormal growth of log files. The node's SQLite cache database returned an "insufficient disk space" error when performing write operations, and the local cache became unavailable.
[0087] When a node detects a write failure to the cache database, it automatically triggers a circuit breaker mechanism: immediately switching the data source for the current and subsequent transactions to the remote database. During the circuit breaker state, all keys required for cryptographic operations by this node are obtained directly from the remote database, the transaction chain is unaffected, and upstream business systems are unaware of the disruption.
[0088] Meanwhile, the system started a background recovery thread, attempting to reopen the SQLite cache database and perform a test read / write operation every 10 seconds. After receiving the disk alert, the operations personnel cleaned up expired log files and freed up disk space three minutes after the failure. On the fifth retry (approximately 50 seconds after the failure), the background recovery thread's test read / write operation succeeded, the system automatically lifted the circuit breaker, switched the data source back to the normal mode with local cache priority, and began gradually resynchronizing the key data from the remote database to the local cache through a scheduled scanning mechanism.
[0089] This case demonstrates that the circuit breaker and automatic recovery mechanism of the present invention achieves a seamless transition in cache layer failure scenarios without the need for manual service restart or manual intervention, significantly improving the robustness and automation level of the financial cryptography service platform under high availability requirements.
[0090] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0091] It should be noted that the specific product or technology names such as financial cryptographic security service platform, SQLite, and Redis mentioned in this invention are for illustrative purposes only and do not constitute a limitation on the scope of application of the technical solution of this invention. Those skilled in the art should understand that the technical solution of this invention is also applicable to other types of embedded caching databases, other types of cryptographic service platforms, and other business scenarios requiring high-performance key synchronization.
[0092] Furthermore, the specific parameter values involved in the embodiments, such as a scan interval of 5 seconds, a sleep cycle of 10 seconds, a UDP retransmission interval of 0.5 seconds, a page cache size of 256MB, and a transaction waiting time of 100ms, are all specific examples of implementing the technical solution of the present invention. Those skilled in the art can make adaptive adjustments according to actual business needs and system environment during the specific implementation process, and these adjustments do not depart from the protection scope of the present invention.
[0093] The technical features described in this invention and their equivalent substitutions are all within the scope of protection claimed by this invention. Any changes, substitutions, or combinations that do not depart from the essential content of the technical solution of this invention should be covered within the scope of protection of this invention.
Claims
1. A key synchronization method for an embedded cache database, characterized in that, include: A two-tier storage architecture is adopted, consisting of a remote database and a local embedded cache database, with the remote database being identified as the single source of facts; The local embedded cache database is embedded in the cryptographic service process as a library file; During the service process startup phase, the key data in the remote database is fully loaded into the local embedded cache database; When reading the key, the local cache is read first; in case of an error, the system switches to the remote database. When updating the key, it is first persisted to the remote database. After successful update, the local cache is updated synchronously, and a broadcast notification using the User Datagram Protocol is sent to other nodes in the cluster. The critical synchronization message is sent at least twice. Build an independent monitoring process to periodically scan for key changes in the remote database and synchronize them to the local cache; The current version and historical version keys are stored in the local cache. When a transaction fails due to a specific error, the current version, historical version and the key in the remote database are used for progressive retry verification in sequence. When a local cache failure is triggered, the circuit breaker is activated, and the remote database is used as the data source. A background recovery thread is started to periodically test the local cache. The circuit breaker is deactivated once the test is successful.
2. The key synchronization method for an embedded cache database according to claim 1, characterized in that, The local embedded cache database uses SQLite and interacts directly with the password service process through a native C language application programming interface.
3. The key synchronization method for an embedded cache database according to claim 1, characterized in that, The specific errors include message verification errors or PIN conversion errors.
4. The key synchronization method for an embedded cache database according to claim 1, characterized in that, The timed scan interval is configured to be 5 seconds, and the scan window is set to the key changes in the remote database within the last 10 seconds.
5. The key synchronization method for an embedded cache database according to claim 1, characterized in that, The test operations for the background recovery thread include reopening the local embedded cache database and performing test read and write operations.
6. The key synchronization method for an embedded cache database according to claim 1, characterized in that, After the circuit breaker is triggered, all transaction processing under the circuit breaker state is directly connected to the remote database, and the test cycle of the background recovery thread is independent of the scan cycle of the timed scan.
7. The key synchronization method for an embedded cache database according to claim 1, characterized in that, The broadcast notification is received using a dedicated receiving service with a multi-process architecture. The total number of processes in this receiving service is configured as the number of platform nodes minus one multiplied by three.
8. The key synchronization method for an embedded cache database according to claim 1, characterized in that, The local embedded cache database uses a memory mapping mechanism to map the database file to the process address space, and uses a write-ahead log mode to ensure the atomicity and durability of transactions.
9. The key synchronization method for an embedded cache database according to claim 1, characterized in that, When a connection to a remote database fails, the occupied shared memory resources are released and the service process is terminated.
10. A key synchronization system for an embedded cache database, characterized in that, include: The data storage architecture module is used to configure a two-tier storage architecture consisting of a remote database and a local embedded cache database, and to identify the remote database as a single source of truth. The local embedded cache database is embedded in the cryptographic service process as a library file; The full preheating module is used to load all key data from the remote database into the local embedded cache database during the service process startup phase. The key reading module is used to prioritize reading the local cache when reading keys, and switch to the remote database in case of an error; The key update module is used to persist the key to the remote database first when updating the key. After success, it synchronously updates the local cache and sends a broadcast notification using the User Datagram Protocol to other nodes in the cluster. The key synchronization message is sent at least twice. The scheduled scanning module is used to build an independent monitoring process to periodically scan for key changes in the remote database and synchronize them to the local cache; The window period transaction compensation module is used to simultaneously retain the current version and historical version keys in the local cache, and to perform progressive retry verification by sequentially using the keys of the current version, historical version and remote database when the transaction fails due to a specific error. The circuit breaker recovery module is used to trigger a circuit breaker when the local cache fails. It directly uses the remote database as the data source and starts a background recovery thread to periodically test the local cache. The circuit breaker is released after the test is successful.