Data interface high-concurrency processing method based on multi-level cache

Abstract

The invention relates to a data interface high-concurrency processing method based on multi-level cache, and belongs to the technical field of computer software. According to the method, an SDK interface receives a user data query request; the request firstly arrives at the local cache, the system queries according to the Key, if the request is hit, the data is directly returned, and the process is ended; if the local cache is not hit, querying a distributed cache Redis; if the distributed cache is hit, returning the data to the user, and asynchronously writing the data back to the local cache for subsequent request use; if the Redis still does not hit, query is executed through the unified data access service; the database query result is backfilled to the distributed cache and the local cache, and a subsequent request can obtain data from the distributed cache and the local cache. Through an automatic adaptation mechanism, data consistency and access correctness among the databases are guaranteed, and the problem of cross-database compatibility is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology, specifically relating to a high-concurrency data interface processing method based on multi-level caching. Background Technology

[0002] With the deepening of digital transformation, various application systems are facing unprecedented pressure from high-concurrency access. As the hub for data interaction between systems, the performance of data interfaces directly determines user experience and business processing capabilities. Currently, common data access solutions mainly suffer from the following shortcomings: Databases become a performance bottleneck: Relational databases (especially domestically produced databases in complex query scenarios) have limited IO processing capabilities and connection pool resources. Under sudden high concurrency requests, a large number of requests directly penetrate to the database, which can easily lead to the exhaustion of the database connection pool, excessive CPU and IO load, resulting in slow response or even service unavailability.

[0003] Inadequate handling of typical caching problems: Existing solutions lack a unified and efficient approach to issues such as cache penetration (querying non-existent data), cache breakdown (expiration of hot keys), and cache avalanche (simultaneous expiration of a large number of keys).

[0004] Therefore, there is an urgent need for a high-concurrency processing technology that can integrate the advantages of multi-level caching and adapt well to domestic database environments to solve the above-mentioned technical pain points. Summary of the Invention (a) Technical problems to be solved The technical problem this invention aims to solve is how to provide a high-concurrency processing method for data interfaces based on multi-level caching, so as to solve the problems of shielding differences between domestic databases, integrating the advantages of multi-level caching, and effectively dealing with cache failure and database bottlenecks in high-concurrency scenarios.

[0005] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a high-concurrency data interface processing method based on multi-level caching. This method is implemented through a three-level caching architecture system, which includes a local cache, a distributed cache, and a unified data access service, and shields the underlying database differences through a unified adaptation layer. The method includes the following steps: Step S11: The SDK interface receives user data query requests; Step S12: The request first arrives at the local cache. The system queries based on the key. If a match is found, the data is returned directly, and the process ends. Step S13: If the local cache is not found, query the distributed cache Redis; Step S14: If the distributed cache is hit, the data is returned to the user and asynchronously written back to the local cache for use in subsequent requests. Step S15: If Redis still fails to find the match, perform a query through the Unified Data Access Service. Step S16: Fill the database query results back into the distributed cache and local cache, so that subsequent requests can retrieve data from the distributed cache and local cache.

[0006] (III) Beneficial Effects This invention proposes a high-concurrency data interface processing method based on multi-level caching. Compared with existing technologies, this invention has the following advantages: The system fully supports domestic databases such as Shenzhou General, Wuhan Dameng, Renmin University Kingbase, and Nanda General, and can be extended to other database types. There's no need to worry about underlying differences, significantly improving development efficiency and system maintainability. Through an automated adaptation mechanism, data consistency and access correctness between databases are ensured, effectively solving cross-database compatibility issues.

[0007] 1. Wide compatibility with domestic databases: Through a unified data change capture and adaptation module (SQL template), it supports mainstream domestic databases, reducing the complexity of system implementation and maintenance; 2. High performance and low latency: The multi-level caching architecture significantly reduces the number of database accesses, improving data response speed and system throughput. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the multi-level caching workflow of the present invention. Figure 2 A schematic diagram of the database adaptation layer implementation. Detailed Implementation

[0009] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0010] This invention relates to the field of computer software technology, particularly distributed computing and high-concurrency data processing technology. Specifically, this invention provides a high-concurrency data interface processing method based on a multi-level caching architecture. This method integrates local caching, distributed remote caching, and various domestic relational databases (including but not limited to DM, Kingbase, GBase, and Shenzhou General), which can significantly improve the throughput of the data interface, reduce response latency, and enhance the stability and availability of the system in high-concurrency scenarios.

[0011] In the context of information technology innovation, the system needs to be compatible with various domestic databases. Different domestic databases differ in SQL syntax, transaction mechanisms, and performance, and there is a lack of a universal, high-performance data access method that can shield the differences between the underlying databases.

[0012] There is a need for a unified data access method that can mask differences between domestic databases, integrate the advantages of multi-level caching, and effectively address cache failures and database bottlenecks in high-concurrency scenarios.

[0013] The core innovations include: 1. Multi-level caching coordination mechanism: Through the collaborative work of local cache and distributed cache, combined with asynchronous update and publish-subscribe pattern, efficient data synchronization and consistency maintenance are achieved.

[0014] 2. Unified adaptation to domestic databases: Adopting object mapping mechanism and dynamic SQL generation technology, it automatically adapts to the differences in data types and SQL syntax of different databases, achieving seamless expansion.

[0015] This invention proposes a high-concurrency data interface processing method based on multi-level caching. This method is implemented through a three-level caching architecture system, which includes a local cache, a distributed cache, and a unified data access service. A unified adaptation layer masks the differences in the underlying database. The specific technical solution is as follows: I. System Composition Local caching: Implemented based on a high-performance on-heap caching library (Caffeine), providing nanosecond-level access speeds for caching extremely hot data.

[0016] Distributed caching: Using a Redis cluster as a centralized cache, the capacity can be horizontally scaled to cache most of the hot data and ensure data consistency.

[0017] Unified Data Access Service: By dynamically generating SQL and mapping data types, it shields the differences between various domestic databases and provides a consistent data access interface.

[0018] Database: Supports multiple domestic databases as persistent storage and final data source.

[0019] II. Data Query Process Step S11: The SDK interface receives user data query requests.

[0020] Step S12: The request first arrives at the local cache (Caffeine). The system queries based on the key. If a match is found, the data is returned directly, and the process ends. This process is a memory access, with a response time in the microsecond range.

[0021] Step S13: If the local cache is not hit, query the distributed cache Redis.

[0022] Step S14: If the distributed cache is hit, the data is returned to the user and asynchronously written back to the local cache for use in subsequent requests.

[0023] Step S15: If Redis still fails to find the match, perform a query through the Unified Data Access Service.

[0024] S151. Define the data model using POJO (Plain Old Java Object) form. Domestic databases (Shenzhou General, Wuhan Dameng, Renmin University Kingbase, and Nanda General) uniformly use the same POJO definition.

[0025] S152. In domestic databases, each inner class uses the @Dialect annotation to implement database-specific type mapping (such as VARCHAR, TEXT, and DATETIME data types). S153. Unify API interface definition and pass POJO objects through API interface calls.

[0026] S154. Configure SQL template configuration files, including Shenzhou general SQL template, Dameng SQL template, Renmin University Kingbase SQL template and Nanjing University general SQL template.

[0027] S155. Select different SQL templates according to the configured database type to generate corresponding dynamic query SQL, especially in handling pagination, functions, and auto-incrementing primary key retrieval (when performing pagination queries, Shenzhou General uses the TOP keyword; Dameng uses the LIMIT and OFFSET keywords; Kingbase uses the LIMIT and OFFSET keywords; and Nanjing University General uses the LIMIT and OFFSET keywords).

[0028] S156. Map POJO object properties to SQL parameters and handle type differences between different databases.

[0029] S157. Based on the result mapping configuration, map the query results back to POJO objects. First, obtain the result set metadata, including the number of columns and column names. Iterate through each column of the result set, finding the corresponding field in the POJO object based on the column name. Then, retrieve the value of that column from the result set. If the POJO object has a corresponding field, perform type conversion (converting the value from the database to a Java type). Finally, set the field value of the POJO object using reflection.

[0030] S158. Encapsulate the results in a uniform format (such as JSON, a standard object, or a custom response) and return it to the application. Include metadata (such as execution status and error messages) to ensure the application receives a consistent interface.

[0031] Step S16: Fill the database query results back into the distributed cache and local cache, so that subsequent requests can retrieve data from the distributed cache and local cache.

[0032] III. Data Update Process Step S21: Perform the update operation. Through the unified data access service interface, route the operation to the correct domestic data source and perform a transactional update.

[0033] Step S22: After the update is successful, immediately delete the corresponding cached data in Redis.

[0034] Step S23: Subsequently, the system sends a cache invalidation notification to the message middleware, the message body of which contains the deleted cache key.

[0035] Step S24: All application instances in the cluster subscribe to the expired topic. Upon receiving the message, each instance asynchronously clears the corresponding data from its local cache.

[0036] Compared with the prior art, the present invention has the following advantages: The system fully supports domestic databases such as Shenzhou General, Wuhan Dameng, Renmin University Kingbase, and Nanda General, and can be extended to other database types. There's no need to worry about underlying differences, significantly improving development efficiency and system maintainability. Through an automated adaptation mechanism, data consistency and access correctness between databases are ensured, effectively solving cross-database compatibility issues.

[0037] 1. Wide compatibility with domestic databases: Through a unified data change capture and adaptation module (SQL template), it supports mainstream domestic databases, reducing the complexity of system implementation and maintenance; 2. High performance and low latency: The multi-level caching architecture significantly reduces the number of database accesses, improving data response speed and system throughput.

[0038] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A high-concurrency data interface processing method based on multi-level caching, characterized in that, This method is implemented through a three-level caching architecture system, which includes a local cache, a distributed cache, and a unified data access service, and shields the differences in the underlying database through a unified adaptation layer. The method includes the following steps: Step S11: The SDK interface receives user data query requests; Step S12: The request first arrives at the local cache. The system queries based on the key. If a match is found, the data is returned directly, and the process ends. Step S13: If the local cache is not found, query the distributed cache Redis; Step S14: If the distributed cache is hit, the data is returned to the user and asynchronously written back to the local cache for use in subsequent requests. Step S15: If Redis still fails to find the match, perform a query through the Unified Data Access Service. Step S16: Fill the database query results back into the distributed cache and local cache, so that subsequent requests can retrieve data from the distributed cache and local cache.

2. The high-concurrency data interface processing method based on multi-level caching as described in claim 1, characterized in that, Local caching: Implemented based on a high-performance on-heap caching library, providing nanosecond-level access speeds and used for caching extremely hot data; Distributed caching: Using a Redis cluster as a centralized cache, the capacity can be horizontally scaled to cache most of the hot data and ensure data consistency; Unified Data Access Service: By dynamically generating SQL and mapping data types, it shields the differences between various domestic databases and provides a consistent data access interface.

3. The high-concurrency data interface processing method based on multi-level caching as described in claim 1, characterized in that, Step S15 includes: S151. Define the data model in POJO form. Domestic databases should uniformly use the same POJO definition. S152. Each inner class in a domestic database uses the @Dialect annotation to achieve database-specific type mapping; S153. Unify API interface definition and pass POJO objects through API interface calls; S154. Configure the SQL template configuration file; S155. Select different SQL templates according to the configured database type to generate corresponding dynamic query SQL; S156. Map POJO object properties to SQL parameters and handle type differences between different databases; S157. Based on the result mapping configuration, map the query results back to POJO objects; S158. Encapsulate the results into a uniform format and return them to the application; including metadata, to ensure that the application receives a consistent interface.

4. The high-concurrency data interface processing method based on multi-level caching as described in claim 3, characterized in that, In S151, the domestic databases include: Shenzhou General, Wuhan Dameng, Renmin University Kingbase, and Nanda General.

5. The high-concurrency data interface processing method based on multi-level caching as described in claim 4, characterized in that, In S154, the SQL templates include: Shenzhou General SQL Template, Dameng SQL Template, Renmin University Kingbase SQL Template and Nanjing University General SQL Template.

6. The high-concurrency data interface processing method based on multi-level caching as described in claim 3, characterized in that, In S152, the specific types include: VARCHAR, TEXT, and DATETIME data types.

7. The high-concurrency data interface processing method based on multi-level caching as described in claim 3, characterized in that, In S155, when processing pagination queries, Shenzhou General uses the TOP keyword; Dameng uses the LIMIT and OFFSET keywords; Kingbase uses the LIMIT and OFFSET keywords; and Nanda General uses the LIMIT and OFFSET keywords.

8. The high-concurrency data interface processing method based on multi-level caching as described in claim 3, characterized in that, In step S157, firstly, the result set metadata is obtained to get the number of columns and column name information; then, each column of the result set is traversed, and the corresponding field in the POJO object is found according to the column name; then, the value of the column is obtained from the result set, and if the POJO object has a corresponding field, a type conversion is performed, that is, the value in the database is converted into a Java type value; finally, the field value of the POJO object is set through reflection.

9. The high-concurrency data interface processing method based on multi-level caching as described in claim 3, characterized in that, In S158, the uniform format is JSON, a standard object, or a custom response.

10. The high-concurrency data interface processing method based on multi-level caching as described in any one of claims 3-9, characterized in that, Step S21: Perform the update operation. Through the unified data access service interface, route the operation to the correct domestic data source and perform a transactional update. Step S22: After the update is successful, immediately delete the corresponding cached data in Redis; Step S23: Subsequently, the system sends a cache invalidation notification to the message middleware, the message body of which contains the deleted cache key; Step S24: All application instances in the cluster have subscribed to the invalidated topic; once a message is received, each instance will asynchronously clear the corresponding data in its local cache.

Images