High-speed information interaction method and system based on three-level cache coordination mechanism

Abstract

This application provides a high-speed information interaction method and system based on a three-level caching coordination mechanism, relating to the field of computer data processing technology. When writing data, the method first acquires the target high-speed information and sends it to a write topic in a Kafka message queue, allocating information partitions within the write topic according to hash ratios. Then, the consumer group of the microservice node pulls the Kafka message queue and generates a unified cache key. Based on this unified cache key, the target high-speed information is then written in parallel to the local second-level cache, the distributed third-level cache, and the database using a microservice thread pool. This method can be based on a layered architecture of a central platform first-level cache, a microservice local second-level cache, and a distributed third-level cache. It uses a thread pool to update high-speed information in parallel, and combined with hash ratio partitioning rules, ensures the sequentiality of events for the same entity, improving the throughput and efficiency of information interaction.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a high-speed information interaction method and system based on a three-level cache collaborative mechanism. Background Technology

[0002] High-speed information, also known as high-concurrency data, refers to real-time data with high concurrency and low latency processing requirements. It can be applied to business scenarios such as logistics tracking, order status changes, and financial information push notifications. For example, real-time data in financial information push notification scenarios requires a peak throughput of 200,000 data points per second and a data consistency greater than 99.99%. For real-time data related to logistics tracking, the query response latency must be less than or equal to 50ms.

[0003] To meet the demands of high-concurrency data interaction, a database and a single-layer remote dictionary server (Redis) caching architecture can be established. During writes, the MySQL database is updated first, followed by a synchronous update to the Redis cache. During queries, the Redis cache is accessed first; if a match is not found, the query retrieves data from the origin MySQL database. For example, e-commerce platforms can employ a database and Redis caching architecture during peak sales periods, using Redis cached keys such as "order:{orderId}" to facilitate data interaction.

[0004] However, the aforementioned caching architecture suffers from throughput bottlenecks due to write path coupling. Furthermore, the lack of a caching hierarchy and scheduling mechanism makes the consistency guarantee mechanism fragile and lacking in compensation capabilities, while coarse-grained scaling leads to resource waste. In addition, the cache update strategy is strongly correlated with database writes, making it impossible to implement an automatic, multi-level write-back mechanism based on data timeliness. This means that queries for historical data still require database access, failing to fully utilize the caching hierarchy to reduce latency and lowering the efficiency of high-speed information exchange. Summary of the Invention

[0005] In view of this, embodiments of this application provide a high-speed information interaction method and system based on a three-level cache coordination mechanism to solve the problem of low efficiency in high-speed information interaction.

[0006] According to a first aspect of this application, a high-speed information interaction method based on a three-level cache coordination mechanism is provided, the method comprising: Acquire target high-speed information, which includes business data generated from external data sources; The target high-speed information is sent to the write topic of the Kafka message queue through the expressive state transfer interface, so as to allocate information partitions in the write topic according to the hash ratio score; Consumer groups of microservice nodes pull the Kafka message queue and generate a unified cache key by parsing the target high-speed information; Based on the unified cache key, the three-level cache write is executed in parallel through a microservice thread pool to write the target high-speed information to the target writing medium, which includes a local second-level cache, a distributed third-level cache, and a database.

[0007] In some embodiments, the method further includes: Obtain information query requests, including user query requests input by the user through the application programming interface gateway; Using the application programming interface gateway, the unified cache key is generated based on the information query request; Multi-level cache queries are performed based on the unified cache key to retrieve the target high-speed information from the target query medium; the target query medium includes a central platform first-level cache, a local second-level cache, a distributed third-level cache, and a database; The target high-speed information is backfilled into the target query medium according to a preset cache order; the preset cache order is the order from the distributed level 3 cache, through the local level 2 cache to the middle platform level 1 cache.

[0008] In some embodiments, performing a multi-level cache query based on the unified cache key to query target high-speed information from the target query medium includes: A first query request is generated based on the unified cache key; In response to the first query request, the target high-speed information is queried from the first-level cache of the middle platform; If a match is found in the first-level cache of the middle platform, the target high-speed information returned by the first-level cache of the middle platform is extracted; If the query fails to find the target high-speed information in the first-level cache of the central platform, the target high-speed information is queried in the local second-level cache according to the unified cache key.

[0009] In some embodiments, querying the target high-speed information in the local secondary cache according to the unified cache key includes: A second query request is generated based on the unified cache key; In response to the second query request, the target high-speed information is queried from the local second-level cache; If a query hits in the local second-level cache, the target high-speed information returned by the local second-level cache is extracted, and the target high-speed information is used to synchronously update the middleware first-level cache; If the query in the local second-level cache is not found, the target high-speed information is queried in the distributed third-level cache according to the unified cache key.

[0010] In some embodiments, querying the target high-speed information in the distributed three-level cache according to the unified cache key includes: A third query request is generated based on the unified cache key; In response to the third query request, the target high-speed information is queried from the distributed three-level cache; If a query hits in the distributed three-level cache, the target high-speed information returned by the distributed three-level cache is extracted, and the target high-speed information is used to synchronously update the middle platform first-level cache and the local second-level cache; If the query in the distributed three-level cache is not found, the target high-speed information is queried in the database according to the unified cache key.

[0011] In some embodiments, querying the target high-speed information in the database according to the unified cache key includes: A fourth query request is generated based on the unified cache key; The fourth query request is sent to the database to query the target high-speed information in the database; If the query in the database is successful, the target high-speed information returned by the database is extracted, and the target high-speed information is used to synchronously update the middle platform first-level cache, the local second-level cache, and the distributed third-level cache. If the query fails to find a match in the database, a query failure message will be generated.

[0012] In some embodiments, the method further includes: Monitor the exception classes during the data writing process; When the detected exception class is detected, a cache rollback is performed to restore the local second-level cache and the distributed third-level cache to the state before the target high-speed information was written. Based on the detected anomaly type, query the failed write event, and the failed write event is associated with the target high-speed information; The failure is written to the target high-speed information associated with the event and delivered to the retry topic of the Kafka message queue; Based on the retry topic, the target high-speed information is rewritten to the L3 cache.

[0013] In some embodiments, after re-performing the L3 cache write on the target high-speed information based on the retry topic, the method further includes: Record the number of failed retries for re-execution of the L3 cache write; When the number of failed retries reaches a preset retry threshold, the failure event is written to a persistent log, which is stored in a log data analysis platform. The persistent log is scanned based on a timed compensation task to identify the failed write event; A three-level cache write is performed on the target high-speed information associated with the failed write event.

[0014] In some embodiments, the method further includes: Calculate the data popularity of the target highway information, whereby the data popularity includes the number of queries to the target highway information per unit time. The popularity threshold is obtained, which includes a first popularity threshold set for the local second-level cache and a second popularity threshold set for the distributed third-level cache; the first popularity threshold is greater than the second popularity threshold. Based on the data popularity, the target high-speed information is synchronized to the cache space associated with the popularity threshold.

[0015] According to a second aspect of this application, a high-speed information interaction system based on a three-level cache coordination mechanism is provided, the system comprising: An external real-time data source module is used to acquire target high-speed information, which includes business data generated by an external data source. The Kafka message queue module is used to send the target high-speed information to the write topic of the Kafka message queue through the expressive state delivery interface, so as to allocate information partitions in the write topic according to the hash ratio score; The microservice node cluster module is used to pull the Kafka message queue using the consumer group of the microservice nodes, and generate a unified cache key by parsing the target high-speed information; The information writing module is used to write the target high-speed information to the target writing medium by performing three-level cache writing in parallel through a microservice thread pool based on the unified cache key. The target writing medium includes a local second-level cache, a distributed third-level cache, and a database.

[0016] According to a third aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described high-speed information interaction method based on a three-level cache coordination mechanism.

[0017] According to a fourth aspect of this application, a storage medium is provided on which a computer program is stored, wherein the program, when executed by a processor, implements the above-described high-speed information interaction method based on a three-level cache coordination mechanism.

[0018] Based on the above technical solution, this application provides a high-speed information interaction method and system based on a three-level cache coordination mechanism. When writing data, the method first acquires the target high-speed information and sends it to a write topic in a Kafka message queue, allocating information partitions within the write topic according to hash ratios. Then, the consumer group of the microservice node pulls the Kafka message queue and generates a unified cache key. Based on the unified cache key, the target high-speed information is written in parallel to the local second-level cache, the distributed third-level cache, and the database using a microservice thread pool. This method can be based on a layered architecture of a central platform first-level cache, a microservice local second-level cache, and a distributed third-level cache. It uses a thread pool to update high-speed information in parallel, and combined with hash ratio partitioning rules, ensures the sequentiality of events for the same entity, improving the throughput and efficiency of information interaction.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram of the information interaction system structure built on a three-level cache coordination mechanism provided in this application embodiment; Figure 2 This is a diagram illustrating the overall architecture of the information interaction system provided in this application embodiment. Figure 3 This is a schematic diagram of the data writing process provided in an embodiment of this application; Figure 4 A flowchart illustrating the asynchronous three-write and compensation mechanism provided in the embodiments of this application; Figure 5 This is a schematic diagram of the data query process provided in the embodiments of this application; Figure 6 This is a schematic diagram of the three-level cache query logic provided in the embodiments of this application; Figure 7 A schematic diagram of the data compensation process provided in the embodiments of this application; Figure 8 This is a schematic diagram of the cache priority scheduling process provided in an embodiment of this application. Detailed Implementation

[0021] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0022] In the embodiments of this application, the high-speed information, also known as high-concurrency data or high-speed data, refers to real-time data with high concurrency and low latency processing requirements, which can be applied to business scenarios such as logistics trajectory, order status change, and financial information push.

[0023] Different business scenarios have corresponding requirements for the efficiency of high-speed information exchange. For example, for real-time data in financial information push scenarios, the peak throughput of market data push needs to reach 200,000 messages per second, and the data consistency must be greater than 99.99%. For real-time data related to logistics trajectories, the response latency for trajectory data queries must be less than or equal to 50ms.

[0024] To meet the demands of high-concurrency data interaction, some implementations can establish a database and a single-layer remote dictionary server (Redis) caching architecture. During writes, the MySQL database is updated first, followed by a synchronous update to the Redis cache. During queries, the Redis cache is accessed first; if a match is not found, the query returns to the origin MySQL database. For example, e-commerce platforms can employ a database and Redis caching architecture during peak sales periods, using Redis cache keys such as "order:{orderId}" to facilitate data interaction. The database can utilize a MySQL master-slave replication architecture, and a time-to-live (TTL) period can be set for the data caching process, such as a uniform 30-minute cache duration.

[0025] In some embodiments, the data caching process can be improved through message queues and a single-layer caching mechanism to alleviate write pressure. By introducing Kafka as a traffic shaping component, external write requests are first sent to a Kafka topic, and then the consumer asynchronously updates the database and Redis. For example, a logistics platform can use a message queue and a single-layer caching mechanism to process the trajectory data of logistics equipment. Under this mechanism, the number of Kafka partitions can be set to 8, the consumption rate of a single partition is approximately 5000 records / second, and the cache hit rate is maintained at 75%-80%.

[0026] The high-speed information interaction methods shown in the above embodiments all have shortcomings, the main one being the throughput bottleneck caused by write path coupling. Since database and cache updates rely on application-layer serial execution or single-threaded asynchronous execution, when the peak write speed exceeds 50,000 records / second, the probability of database connection pool exhaustion is ≥30%, and I / O wait time accounts for over 60%, failing to meet the high-frequency write requirements of 200,000 records / second for securities information and 150,000 records / second for logistics tracing.

[0027] Furthermore, caching layers and scheduling mechanisms are lacking. The information interaction method described in the above embodiments supports a maximum of two-layer caching and lacks global scheduling capabilities for a first-level cache in the middle platform layer. This leads to the problem of different microservices querying the same order status, with cross-service duplicate queries accounting for over 40%. Moreover, hot data is not prioritized for local caching; 30% of high-frequency queries require accessing Redis over the network, adding an extra 50-100ms and failing to meet the industry requirement of query response latency ≤50ms.

[0028] The information interaction methods described in the above embodiments also suffer from the drawback of a fragile consistency guarantee mechanism and a lack of compensation capabilities. If the database write succeeds but the Redis update fails, the dirty data retention time is ≥ TTL (30 minutes), and the inconsistency probability is ≥ 5%. Furthermore, due to the lack of a failure rollback mechanism, when the database write fails but the cache has been updated, the data persistence inconsistency continues until the cache expires, and there is no automatic repair method, which fails to meet the requirement of an inconsistency probability of <0.01% in financial scenarios.

[0029] There is also the problem of resource waste caused by coarse expansion granularity. That is, due to the tight coupling of the collection, processing and storage modules in the information interaction process, such as the application service being responsible for both writing and querying, when the query bottleneck occurs, the application service needs to be expanded as a whole. However, the resource utilization rate of the writing module is only 30%, resulting in more than 50% waste of computing resources.

[0030] Furthermore, because the cache update strategy is strongly correlated with database writes, an automatic, multi-level write-back mechanism based on data timeliness cannot be implemented. This means that queries for historical data still require database access, thus failing to fully utilize cache layers to reduce latency. When a single write fails, the lack of reverse data repair capabilities across cache layers can lead to inconsistencies persisting for extended periods, reducing the efficiency of high-speed information exchange.

[0031] To address the issue of low efficiency in high-speed information exchange, this application provides a high-speed information exchange method based on a three-level caching coordination mechanism in some embodiments. This method enables highly efficient high-speed information exchange based on this mechanism. The three-level caching coordination mechanism employs a read-write separation data update approach for high-speed information, asynchronously writing external data to the local cache, a distributed in-memory database (Redis), and a MySQL database simultaneously, forming a three-write mechanism. This ensures the real-time performance of the locally cached data while also preserving the real-time performance of the Redis and MySQL data. The local cache has a limited caching time; if the queried historical data exceeds the local cache time t1, the locally cached data is written back from Redis. Similarly, the Redis cache has a limited caching time; if the queried historical data exceeds the Redis cache time t2, the Redis data is written back from MySQL, ensuring both real-time and historical data requirements are met simultaneously.

[0032] For example, such as Figure 1 As shown, the information interaction system built on the three-level caching collaboration mechanism can include six core modules: external real-time data source module, Kafka message queue module, microservice node cluster module, distributed caching module, database module, and monitoring and maintenance module.

[0033] The core function of external real-time data sources is to generate high-frequency real-time data, such as logistics GPS data, stock market data, and stock market information. Hardware parameters can be deployed through logistics terminals based on 4G or 5G networks, exchange interface servers / market data and information servers, and data communication can be achieved based on the corresponding communication protocols of Representational State Transfer Application Programming Interface (REST API) or Message Queuing Telemetry Transport (MQTT).

[0034] The core function of the Kafka message queue module is to receive and buffer write requests, partition them by hash according to business entity identifiers, and ensure the order of events. It is deployed through a 3-node cluster, with each node equipped with 16 cores and 32GB of RAM and a 1TB solid-state drive (SSD), and communicates data based on the Kafka protocol.

[0035] The core functions of a microservice node cluster are to execute asynchronous three-way writes, three-level cache queries, and compensation logic, which can include a controller, a service, and a local cache unit. A 5-node cluster, with each node having 8 cores and 16GB of RAM and 8GB of JVM heap memory, is used for hardware deployment. Data communication is conducted via gRemote Procedure Calls (GRPC) or Hypertext Transfer Protocol (HTTP) 2.0.

[0036] The core function of the distributed caching module is to store cached data shared across nodes and support master-slave replication. It utilizes a Redis Cluster (3 masters, 3 slaves), with each node having 8 cores and 32GB of RAM, and communicates data based on the Redis protocol.

[0037] The core function of the database module is data persistence, supporting master-slave synchronization and disaster recovery. It uses a MySQL master-slave setup (1 master, 2 slaves), with each node having 16 cores and 64GB of RAM. Data communication is performed according to the Java Database Connectivity (JDBC) protocol.

[0038] The core functions of the monitoring and maintenance module are to collect metrics (QPS, latency, consistency rate), support automatic scaling, and perform log analysis. It uses Prometheus, Grafana, and the log data analysis platform (Elasticsearch, Logstash, Kibana, ELK), with a 2-node, 8-core, 16-GB hardware deployment, and communicates data via HTTP or HTTPS protocols.

[0039] When an information exchange system interacts with information, it can form different data flows based on data writing, data querying, and data monitoring. For example, the data writing process can form a write data flow that runs through the data source, Kafka, microservices, and three-tier storage; the data query process can form a query data flow that runs through users, API gateways, three-tier caches, and databases; the data processing process can form a monitoring data flow that connects various modules with Prometheus; and the service registration process can form a registration data flow that connects microservices with the registry center.

[0040] Based on a three-level caching coordination mechanism, a three-level caching coordination and scheduling architecture can be formed, including a central platform first-level cache, a microservice local second-level cache, and a distributed third-level cache. For example... Figure 2As shown, the first-level cache of the middle platform can be deployed on the Redis cluster of the business middle platform and supports cross-service sharing; the microservice local second-level cache can be a high-performance local caching library such as Caffeine, residing in application memory; the distributed third-level cache can be deployed based on Redis Cluster and supports cross-node sharing. As a layered architecture, the three-level cache collaborative scheduling architecture can alleviate the problems of insufficient cache layers and duplicate cross-service queries through unified cache key rules and dynamic priority scheduling mechanisms. Compared with the two-level cache architecture, the three-level cache collaborative scheduling architecture can improve the cache hit rate by 20% and reduce cross-service queries by 40%.

[0041] It's important to note that the three-level cache collaborative scheduling architecture is not simply adding a cache layer to a two-level cache architecture. Instead, it achieves efficient collaboration among three cache layers—a central platform first-level cache, a microservice local second-level cache, and a distributed third-level cache—each with distinct performance, scope, and lifecycle. This avoids data inconsistencies or scheduling overhead offsetting performance gains. Furthermore, a unified cache key rule ensures the uniqueness and traceability of data identifiers across the three levels of cache, laying the foundation for collaborative scheduling. A dynamic priority scheduling algorithm based on access frequency intelligently migrates data between the three levels of cache, achieving optimal utilization of global cache resources.

[0042] The high-speed information interaction method based on a three-level caching coordination mechanism can be applied to electronic devices with data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, this application embodiment uses an electronic device as the execution subject of the method. It should be understood that the method can also be applied to other types of execution subjects, which will not be shown in detail in this application embodiment.

[0043] High-speed information exchange based on a three-level caching coordination mechanism can include key processes such as data writing, data querying, and data compensation. For example... Figure 3 As shown, during data writing, data can be sent to Kafka from external information data sources, and consumed by microservices to generate cache keys. This allows the thread pool to perform three write operations in parallel: writing to the local cache, writing to the distributed cache, and writing to the TDSQL (MySQL) database. This method includes: S101, Obtain target high-speed information.

[0044] When writing data, the data to be written can be obtained first, i.e., the target high-speed information can be obtained. This target high-speed information includes business data generated by an external data source. In some embodiments, to obtain the target high-speed information, an information retrieval request can be generated in response to a user-inputted data write command and sent to the external data source. This triggers the external data source to respond to the information retrieval request and provide feedback data. The target high-speed information is then obtained by receiving the feedback data.

[0045] For the data writing process, electronic devices can first access data, that is, generate data by connecting to an external data source to obtain target high-speed information. For example, if the target high-speed information is the logistics trajectory in a logistics system, then the logistics system acts as the external data source. After the electronic device connects to the logistics system, it can send a data acquisition request to the logistics system to obtain the logistics trajectory data. {vehicleId: "× A12345", lat:39.9, lng:116.4, ts:1700000000}; Wherein, vehicleId represents the logistics equipment number, i.e., the vehicle with license plate number "× A12345"; lat represents latitude; lng represents longitude, i.e., lat:39.9, lng:116.4 indicates that the current logistics equipment is located at (39.9, 116.4); ts represents timestamp, so ts:1700000000 represents 22:13:20 on November 14, 2023.

[0046] S102. Through the expressive state transfer interface, the target high-speed information is sent to the write topic of the Kafka message queue, so that the information partitions are allocated in the write topic according to the hash ratio score.

[0047] After obtaining the target high-speed information, it can be partitioned according to a hash-based proportional score partitioning rule. Specifically, the target high-speed information is sent to a write topic in the Kafka message queue via a descriptive state delivery interface, where information partitions are allocated according to hash-based proportional scores within the write topic. The number of partitions can be twice the number of business entity types.

[0048] Among them, the Representational State Transmission Interface (REST API) is an API designed based on the REST architectural style. As a software architectural style, REST can leverage the characteristics of the HTTP protocol to operate on resources on the network as unique identifiers (URIs) through methods such as GET, POST, PUT, and DELETE, and exchange data through standardized methods such as JSON and XML, thereby achieving a stateless, cacheable, and unified API design specification.

[0049] In Kafka, a write topic is a named message stream category or data pipeline that has the business meaning of writing data. For example, a write topic could be "logistics-track". As a write topic related to logistics tracking, logistics-track can send all data to be written for business events related to logistics tracking as messages to this topic. Such messages include "order shipped", "package arrived at sorting center", "in delivery", and "delivered".

[0050] Hash ratio score, calculated using a hash function, determines the number of partitions and is used to allocate partitions while ensuring even data distribution and key order preservation. As a partitioning strategy in distributed systems like Kafka, allocating information partitions according to hash ratio score ensures that all messages with the same ID are sent to the same partition, thus guaranteeing local ordering of data for the same vehicle. Simultaneously, the hash function distributes ID data as evenly as possible across all partitions, achieving load balancing.

[0051] For example, such as Figure 4 As shown, after obtaining target highway information such as logistics trajectory, the logistics trajectory message is sent to a specified topic in Kafka, such as "logistics-track", via a REST API, and partitions are allocated according to hash(vehicleId)%partition number. When allocating partitions, the partition key is determined first to explicitly use vehicleId (logistics equipment number) as the input for partition calculation. Then, a hash function is selected to convert the string-type vehicleId into a definite, uniformly distributed integer. Next, the hash value is processed to be positive, i.e., the hash value is converted to a positive number through bitwise operations while maintaining its distribution. Then, a modulo operation is performed to obtain an integer in the range [0, total number of partitions - 1], which is the target partition number. Finally, the calculated partition key is used as a parameter to send the logistics trajectory message to the specified partition. For example, 8 partitions are used to ensure the sequential tracking of the same vehicle's trajectory.

[0052] S103. Use the consumer group of the microservice node to pull the Kafka message queue and generate a unified cache key by parsing the target high-speed information.

[0053] After sending the messages corresponding to the high-speed information to the Kafka message queue according to the partition, message consumption can be triggered. That is, the consumer group of the microservice node pulls the message data from the Kafka queue, parses the message data, and generates a unified cache key.

[0054] In Kafka, a consumer group is a logical concept consisting of one or more consumer instances, where the number of consumer instances equals the number of partitions. These consumer instances collaborate to consume messages from one or more topics.

[0055] A unified cache key is an identification information generated using unified cache key rules. It ensures the uniqueness and traceability of data identifiers within the three-level cache, laying the foundation for collaborative scheduling. In some embodiments, the unified cache key can support version control to avoid dirty reads. Specifically, the unified cache key is a cache key generated based on business type information, business entity identifier, and version number. As a data identifier, the unified cache key can be represented in the following format: Cache key = Business type + ":" + Business entity identifier + ":" + Version number; The version number can be determined based on the timestamp and data digest in the high-speed information, such as version number = md5(timestamp + data digest), to ensure that different data versions of the same entity are unique. For example, if the timestamp of logistics track data is ts=1700000000 and the data digest is md5(lat+lng)=abc123, then the corresponding version number is md5(1700000000+abc123)=def456. Therefore, the unified cache key generated based on the above information is "track: × A12345:def456".

[0056] S104. Based on a unified cache key, the three-level cache write is executed in parallel through a microservice thread pool to write the target high-speed information to the target writing medium.

[0057] After pulling the message queue and generating a unified cache key, data writing can be performed based on the unified cache key to overwrite the three-level cache. That is, the three-level cache writing is executed in parallel through the microservice thread pool to write the target high-speed information to the target writing medium, wherein the target writing medium includes the local second-level cache, the distributed third-level cache and the database.

[0058] When writing data, the write path can be decoupled based on the Kafka message queue, and a three-write mode can be adopted to update the local cache, distributed cache, and database in parallel using a thread pool. In the parallel three-write mode, the microservice can execute three cache write operations in parallel through the thread pool: writing to the local second-level cache, writing to the distributed third-level cache, and writing to the database.

[0059] When writing to the local second-level cache, the corresponding write parameters and write code can be configured based on the cache format of the local second-level cache. Taking the local second-level cache based on Caffeine as an example, the maximum size (maximumSize) can be configured to 10000, i.e., maximumSize=10000, and the write expiration time (expireAfterWrite) can be set to 5 minutes, i.e., expireAfterWrite=5min.

[0060] Based on the configured write parameters, the write program code "caffeineCache.put("track:×A12345:v1", trackData)" can be executed to write the logistics track message corresponding to "track:×A12345:v1" to the local second-level cache (Caffeine).

[0061] When writing to a distributed three-level cache, a distributed cache data structure can be used for high-speed information caching. Taking a Redis-based distributed three-level cache as an example, the Redis-based distributed three-level cache uses a hash data structure for storage. Therefore, the write parameters can be set as "key=track:×A12345:v1, field=lat / lng / ts, expire=30 minutes". By executing the code logic "redisTemplate.opsForHash().putAll("track:×A12345:v1", trackMap); redisTemplate.expire("track:×A12345:v1", 30, TimeUnit.MINUTES)", the logistics track message corresponding to "track:×A12345:v1" will be written to the distributed three-level cache.

[0062] For writing to a database, the corresponding write procedure can be executed based on the database type. Taking a MySQL database as an example, an asynchronous SQL insert can be performed (insert into track(vehicle_id, latitude, longitude, time) values()), and the asynchronous operation can be achieved using the Spring Async annotation. The corresponding code logic is: @Async; public CompletableFuture <boolean>saveToDb(TrackData trackData) {returnCompletableFuture.supplyAsync(()->{jdbcTemplate.update("insert into track(...)", trackData.getVehicleId(),...); return true;})}.

[0063] After writing the target high-speed information to the target writing medium through parallel three-level cache writing, the writing result can also be verified. If the database writing is successful, i.e. "CompletableFuture returns true", then an acknowledgment character (ACK) is sent to Kafka; if the database writing fails, i.e. "CompletableFuture throws an exception", then compensation logic can be executed.

[0064] The following example illustrates the high-speed information interaction method shown in the above embodiments, using a complete logistics track data writing process. When performing high-speed information interaction, deployment parameters can be configured first. For the Kafka mechanism, the writing topic "logistics-track" can be set, with 8 partitions, 2 replicas, and a message retention time of 24 hours. For the second-level cache (Caffeine), maximumSize=10000, expireAfterWrite=5 minutes, and an initial capacity of 5000. For the third-level cache (Redis), a hash data structure is set, expire=30 minutes, and master-slave synchronization latency ≤100ms. For the MySQL database, the table track(vehicle_id varchar (20), latitude double, longitude double, time bigint) can be set, with an index consisting of a composite index of vehicle_id and time.

[0065] Based on the deployed parameters, the data writing process is then executed. The logistics vehicle terminal (× A12345) sends trajectory data to the Kafka message queue every 5 seconds. After consumption by the microservice, a key "track: × A12345:def456" is generated based on the key generation rules. This ensures that even if the same vehicle continuously reports multiple trajectories within a very short period, different version numbers can be generated due to differences in timestamps and content. This allows for parallel and accurate storage and location in a three-level cache, which is the foundation for supporting asynchronous three-way writes. The data is then written in parallel to Caffeine, Redis, and MySQL.

[0066] By applying the technical solutions of the above embodiments, the high-speed information interaction method based on the three-level cache collaboration mechanism described in the above embodiments can realize an asynchronous three-write parallel writing mechanism based on the three-level cache collaboration architecture. It decouples the write path based on the Kafka message queue and adopts a three-write mode of parallel updating the local cache, distributed cache, and database using a thread pool, greatly improving the throughput of written data. Simultaneously, the partitioning rule of hashing the partition number of the business entity identifier ensures the sequentiality of events for the same entity, breaking through the write throughput bottleneck. Furthermore, combined with the unified cache key rule in the three-level cache collaboration architecture, a version number containing a data digest is generated as a unified cache key, enabling query requests to accurately locate the latest version of data using the key even with minimal latency in parallel writing, avoiding dirty reads. Compared to the dual-caching scheme, the write efficiency is improved by 10 times.

[0067] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a high-speed information interaction method based on a three-level cache coordination mechanism for implementing data querying. For example... Figure 5 As shown, the method includes: S201, Obtain information query request; S202. Use the application programming interface gateway to generate a unified cache key based on the information query request; S203. Perform multi-level cache queries based on a unified cache key to retrieve target high-speed information from the target query medium; S204. Fill the target high-speed information back into the target query medium according to the preset cache order.

[0068] To perform data queries, an information query request is first obtained, which includes a user query request input by the user through an application programming interface (API) gateway. Then, in response to the information query request, the API gateway generates a unified cache key based on the query request. The unified cache key can be generated in the same way as in the above embodiments, both based on unified cache key rules.

[0069] After generating a unified cache key, multi-level cache queries are performed based on this key to retrieve target high-speed information from the target query medium. The target query medium includes a central platform first-level cache, a local second-level cache, a distributed third-level cache, and a database. For example, when performing a data query, the user's information query request first enters the API gateway and queries the first-level cache (central platform Redis). If a match is found, the retrieved data is returned. If no match is found, the second-level microservice cache (Caffeine) is queried. If a match is found, the corresponding data is returned and the first-level cache is updated. If no match is found, the third-level cache (RedisCluster) is queried. If a match is found, the corresponding data is returned and the first and second-level caches are updated.

[0070] Then, the target high-speed information is backfilled to the target query medium according to the preset cache order. The preset cache order is from the distributed three-level cache, through the local two-level cache, to the central platform's first-level cache. That is, the time range for each step is determined by annotation requirements, such as the query time for the first and second-level caches being ≤1ms, and the cache TTL is set, such as 10 seconds for the first-level cache. The backfilling order also needs to be set so that when the third-level cache misses, the MySQL database can be queried, the corresponding data returned, and the data backfilled according to the order of third-level cache, second-level cache, and first-level cache.

[0071] For example, in the data query process, when a user queries the logistics trajectory message of "×A12345", the system can first query the first-level cache of the middle platform (Redis, expire=10 seconds). If the query does not find the data, it will query the local Caffeine of the microservice. If the query finds the data, it will return the data, taking 3ms.

[0072] like Figure 6 As shown, in some embodiments, when performing a multi-level cache query based on a unified cache key to retrieve target highway information from the target query medium, a first query request can be generated based on the unified cache key, and in response to the first query request, the target highway information can be queried in the first-level cache of the middle platform. If the query hits in the first-level cache of the middle platform, the target highway information returned by the first-level cache of the middle platform is retrieved; if the query misses in the first-level cache of the middle platform, the target highway information is queried in the local second-level cache based on the unified cache key.

[0073] When querying target highway information in the local second-level cache based on the unified cache key, a second query request can be generated based on the unified cache key. In response to the second query request, the target highway information can be queried in the local second-level cache. If the query finds a match in the local second-level cache, the target highway information returned by the local second-level cache is retrieved, and the first-level cache of the middle platform is synchronously updated using the target highway information. If the query does not find a match in the local second-level cache, the target highway information is queried in the distributed third-level cache based on the unified cache key.

[0074] Similarly, when querying target highway information in the distributed three-level cache based on the unified cache key, a third query request can be generated based on the unified cache key. In response to this third query request, the target highway information can be queried in the distributed three-level cache. If the query finds a match in the distributed three-level cache, the target highway information returned by the distributed three-level cache is retrieved, and the middleware first-level cache and local second-level cache are synchronously updated using the target highway information. If the query does not find a match in the distributed three-level cache, the target highway information is queried in the database based on the unified cache key.

[0075] When querying the target highway information in the database based on the unified cache key, a fourth query request can be generated based on the unified cache key and sent to the database to retrieve the target highway information. If the query finds a match in the database, the target highway information returned by the database is extracted, and the middleware first-level cache, local second-level cache, and distributed third-level cache are synchronously updated using the target highway information; if the query does not find a match in the database, a query failure message is generated.

[0076] For example, when performing a data query, you can first obtain the user's query request through an access request, such as "GET / track / ×A12345". Accessing through the API gateway, the gateway generates a cache key track:×A12345:v1.

[0077] Then, based on the three-level cache penetration query, the first level cache query is performed, that is, the query of the middle platform first level cache (Redis) is performed, with expire=10 seconds configured. If the query hits, that is, cache.getIfPresent(key) != null, the data is returned directly, with a time of ≤1ms; if the query misses, the second level cache query is entered.

[0078] When performing a second-level cache lookup, the microservice's local Caffeine cache can be queried. If the second-level cache lookup is successful, the data is returned and the first-level cache is updated synchronously, i.e., cache.put(key, data), with a time consumption of ≤2ms. If the second-level cache lookup fails, the third-level cache lookup is initiated.

[0079] When performing a level 3 cache query, you can query Redis Cluster. If the level 3 cache query hits, the data is returned and the level 1 and level 2 caches are updated synchronously, with a time of ≤5ms. If the level 3 cache query misses, the database query is performed.

[0080] When performing a database query, you can query MySQL (select...) From track where vehicle_id=?order by time desc limit 1). After retrieving data through a database query, cache backfilling can be performed, that is, the cache can be backfilled in the order of "level 3 cache - level 2 cache - level 1 cache", and the corresponding TTL is set, with a time consumption of ≤50ms, which is used to trigger the generation of failure prompt information when the cache is not hit.

[0081] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a high-speed information interaction method based on a three-level cache coordination mechanism for data compensation. For example... Figure 7 As shown, the method includes: S301, Monitor the exception classes during the data writing process; S302. When the detected exception class is detected, perform a cache rollback to restore the local second-level cache and the distributed third-level cache to the state before the target high-speed information was written. S303, Write an event if the query fails based on the detected anomaly type; S304. Write the failure to the target high-speed information associated with the event and deliver it to the retry topic of the Kafka message queue; S305, Re-execute the L3 cache write to the target high-speed information based on the retry topic.

[0082] To perform data compensation, you can first set the compensation task parameters, namely the labeling requirements, such as the number of parallel threads, the number of retries and the interval, and the log storage location (ELK). For example, the core thread count is 4, the number of retries is 3, and the retry interval is 1, 2, and 4 seconds to achieve exponential backoff, and the scheduled repair time is set to 2:00 AM every day.

[0083] Data compensation can then be performed based on the compensation task parameters. The data compensation process may include rolling back the two-level cache, retrying writes, writing to logs if retry fails, and a scheduled task scanning and repairing the logs. Therefore, we can first listen for checked exception classes during the data write process. When a checked exception class is detected, we can perform a cache rollback to restore the local second-level cache and the distributed third-level cache to their state before the target high-speed information was written. Then, we can query the failed write events based on the checked exception classes, where the failed write events are associated with the target high-speed information. Finally, we can deliver the target high-speed information associated with the failed write events to the retry topic in the Kafka message queue to re-execute the third-level cache write based on the retry topic.

[0084] In some embodiments, when re-performing the Level 3 cache write to the target high-speed information based on a retry topic, preset retry parameters can be obtained first, including the number of retries and the retry interval. Information partitions are then allocated in the retry topic according to a hash ratio score. Next, according to the preset retry parameters, the consumer group of the microservice node pulls the Kafka message queue. Then, the unified cache key of the target high-speed information is obtained, and based on this unified cache key, the Level 3 cache write is performed in parallel through a microservice thread pool to write the target high-speed information to the target writing medium.

[0085] For example, if a MySQL write fails, the system will trigger a compensation mechanism tightly coupled with the three-level cache structure and asynchronous write process. This compensation mechanism can roll back the cache and retry three times, logging any failures. Therefore, during data compensation, an immediate rollback can be performed first. When a database write fails, i.e., when a checked exception class of SQLExceptions is caught by listening for checked exception classes, the microservice immediately executes a cache rollback operation.

[0086] Cache rollback operations can include rolling back the second-level cache and rolling back the third-level cache. To roll back the second-level cache, you can execute `caffeineCache.invalidate("track:×A12345:v1")`. To roll back the third-level cache, you can execute `redisTemplate.delete("track:×A12345:v1")`.

[0087] Cache rollback operations can be based on a retry mechanism, which involves resubmitting failed events to a Kafka retry topic, such as the "logistics-track-retry" topic, and setting the number of retries (maxRetry), such as maxRetry=3, with a retry interval of 1 second.

[0088] If the write operation still fails after performing an immediate cache rollback operation based on the retry mechanism, a scheduled repair can be triggered. In some embodiments, after re-performing the L3 cache write to the target high-speed information based on the retry topic, the number of retry failures can be recorded, and when the number of retry failures reaches a preset retry threshold, the failed write event is written to the persistent log. The persistent log is stored on a log data analysis platform. Then, a scheduled compensation task scans the persistent log to identify failed write events and performs a L3 cache write on the target high-speed information associated with the failed write event.

[0089] For example, according to the scheduled repair mechanism, if three retries fail, the event is written to a persistent log (ELK storage), and a scheduled compensation task is executed at 2 AM every day to scan the log and perform the following logic: using "List <failedevent>The `events = logService.getUnSuccessEvents()` function retrieves events from the log that were not successfully written to the database. Then, it re-executes the three-write operation by executing the following: `for (FailedEvent event : events) {if (!dbService.checkExists(event.getVehicleId(), event.getTs())) { executeThreeWrite(event.getData())}}`.

[0090] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a high-speed information interaction method based on a three-level cache coordination mechanism for implementing cache priority scheduling. For example... Figure 8 As shown, the method includes: S401, Calculate the data heat of the target high-speed information; S402, Obtain the heat threshold; S403. Based on data popularity, synchronize the target high-speed information to the cache space associated with the popularity threshold.

[0091] Cache priority scheduling dynamically keeps hot data in the local second-level cache. To implement cache priority scheduling, the data popularity of the target highway information can be calculated first. This data popularity includes the number of queries to the target highway information per unit time.

[0092] For example, data popularity can be calculated by the number of queries in the last 5 minutes. To calculate data popularity, you can execute "int heat=queryCounter.getCount(key, 5, TimeUnit.MINUTES)" to calculate the number of queries in the last 5 minutes.

[0093] After calculating the data popularity of the target high-speed information, a popularity threshold can be obtained. This popularity threshold includes a first popularity threshold set for the local second-level cache and a second popularity threshold set for the distributed third-level cache. Furthermore, the first popularity threshold is greater than the second popularity threshold. For example, for the local second-level cache, a popularity threshold of 100 can be set to keep data exceeding the popularity threshold in the local cache.

[0094] Then, based on the data popularity, the target high-speed information is synchronized to the cache space associated with the popularity threshold. For example, by executing the command "if (heat>100&&!caffeineCache.asMap().containsKey(key)) {Object data =redisTemplate.opsForValue().get(key); caffeineCache.put(key, data)", hot data is synchronized from the third-level cache to the second-level cache.

[0095] For example, if the target highway information is queried 120 times in the past 5 minutes, the target highway information can be marked as hot data according to the dynamic priority scheduling algorithm, and the target highway information can be synchronized from the third-level cache to the second-level cache. This realizes the scheduling strategy of prioritizing access to the hottest level for queries, and reduces latency through the collaboration between the third-level cache and the scheduling algorithm.

[0096] The following example illustrates the high-speed information interaction method shown in the above embodiments using a complete securities information push process. When pushing securities information, parameters can be deployed first, including setting the Kafka mechanism to include the "stock-news" topic, partition number = 16, and message retention time = 1 hour. For the second-level cache (Caffeine), maximumSize = 50000 can be set to accommodate larger information data volumes, and expireAfterWrite = 1 minute to accommodate more timely information. For the third-level cache (Redis), a String data structure is deployed to store JSON strings, with expire = 30 seconds to accommodate information with higher update frequencies. For the MySQL database, a table stock_news(stock_code varchar (10), content text, publish_time bigint) is deployed, with index = stock_code + publish_time. For the compensation task, the number of retries is set to 5, the retry interval to 500ms, and the scheduled repair time to be executed once per hour.

[0097] During the information exchange process, for data writing, the exchange pushes individual stock information every 100ms, such as {stockCode:"600000", content:"...", publishTime:1700000000100}, to the "stock-news" topic in Kafka. The microservice generates the key "news:600000:ghi789" and writes it in parallel to Caffeine, Redis, and MySQL.

[0098] Due to extremely high data timeliness requirements, the TTL settings for each level of cache are significantly shortened (30 seconds for Redis) to reflect collaborative adaptability. By adjusting collaborative strategy parameters such as TTL, the same architecture can perfectly adapt to the stringent requirements of different business scenarios. The asynchronous three-write mechanism ensures the rapid ingestion of high-frequency data, while the combination of short TTL and three-level caching guarantees data real-time performance and effectively withstands query surges through the multi-level structure. If a MySQL write fails, the cache is immediately rolled back and retried 5 times, making the configurability of the compensation mechanism specifically adapted to the higher consistency requirements of the business scenario (finance).

[0099] For data queries, when a user queries the latest information for "600000", the system first checks the first-level cache of the middle platform (expire=5 seconds). If the query is successful (it has been queried 80 times within 1 second), the data is returned directly, taking 1ms. If the query is unsuccessful, the system checks the Redis cache and MySQL, and then refills the cache.

[0100] By applying the technical solutions of the above embodiments, the high-speed information interaction method based on the three-level cache coordination mechanism described in the above embodiments can, through a two-layer compensation logic of failure compensation and consistency repair, immediately call the cache deletion interface when a database write fails, roll back the updated two-level cache, and trigger multiple retries. After a retry failure, a timed compensation task based on Quartz scheduling scans the persistent logs stored in ELK and re-executes the three-write operation on the data not yet written to the database. Through the two-layer compensation mechanism of failure compensation and consistency repair, the probability of inconsistency can be reduced from 5% to below 0.01%, achieving a consistency closed loop.

[0101] The two-tiered compensation logic of retry rollback and timed repair can also address the potential data inconsistency risks brought about by asynchronous three-way writes and complex caching layers, resolving the problem of single-write failures. Furthermore, it constructs a final data consistency repair barrier spanning three levels of storage: local cache, distributed cache, and database. The two-tiered design of instant rollback and timed repair maps to the storage structure hierarchy, ensuring the system has self-healing capabilities in the event of a failure at any level. Through the tight integration of these mechanisms, a complete and robust data processing system can be formed.

[0102] A 72-hour comparative test was conducted in a real business environment, including three types of test environments: "basic configuration", "high load configuration" and "lightweight configuration", to verify the stability of the technology's effectiveness.

[0103] For basic configuration testing, the test hardware used was Kafka (3 nodes 16C32G), microservices (5 nodes 8C16G), Redis (3 masters 3 slaves 8C32G), and MySQL (1 master 2 slaves 16C64G) to process logistics trajectory data from 10,000 logistics vehicles, uploading one trajectory every 5 seconds, with a peak write rate of 2,000 records / second and a query QPS of 5,000.

[0104] In high-load business scenarios, the test hardware used was Kafka (6 nodes, 32 cores, 64GB RAM), microservices (10 nodes, 16 cores, 32GB RAM), Redis (6 masters, 6 slaves, 16 cores, 64GB RAM), and MySQL (2 masters, 4 slaves, 32 cores, 128GB RAM). It pushed market data for 5000 individual stocks, one quote every 100ms, with a peak write speed of 50,000 records / second and a query QPS of 50,000.

[0105] In a lightweight configuration business scenario, we tested the hardware using Kafka (1 node 8C16G), microservices (2 nodes 4C8G), Redis (1 master 1 slave 4C8G), and MySQL (1 master 1 slave 8C16G) to synchronize the order status of a small e-commerce business with 1,000 merchants, updating 1 order every 1 second, with a peak write rate of 1,000 records / second and a query QPS of 1,000.

[0106] The test results are shown in Table 1. Table 1. Comparison of Test Results;

[0107] Based on the above test results, the proposed method achieves significant optimization of core indicators in three hardware environments (basic configuration, high-load configuration, and lightweight configuration) and three business scenarios (logistics, securities, and e-commerce). Specifically, the write throughput reaches a maximum of 120,000 records / second, the P99 latency for high-frequency queries is as low as 3ms, and the data inconsistency probability remains consistently <0.01%, fully meeting the needs of businesses of different scales. Furthermore, resource utilization consistently remains above 78%, demonstrating the strong versatility and stability of the technical solution and fully achieving the preset technical objectives.

[0108] In some embodiments, as a specific implementation of the high-speed information interaction method based on a three-level cache coordination mechanism described in the above embodiments, some embodiments of this application also provide a high-speed information interaction system based on a three-level cache coordination mechanism, such as... Figure 1 As shown, the system includes: An external real-time data source module is used to acquire target high-speed information, which includes business data generated by an external data source. The Kafka message queue module is used to send the target high-speed information to the write topic of the Kafka message queue through the expressive state delivery interface, so as to allocate information partitions in the write topic according to the hash ratio score; The microservice node cluster module is used to pull the Kafka message queue using the consumer group of the microservice nodes, and generate a unified cache key by parsing the target high-speed information; The information writing module is used to write the target high-speed information to the target writing medium by performing three-level cache writing in parallel through a microservice thread pool based on the unified cache key. The target writing medium includes a local second-level cache, a distributed third-level cache, and a database.

[0109] A modular microservice architecture allows the system to be divided into an external real-time data source module, a Kafka message queue module, a microservice node cluster module, a distributed cache module, a database module, and a monitoring and maintenance module. The external real-time data source module handles message access, enabling Kafka consumption and parsing. The microservice node cluster module, distributed cache module, and database module perform cache scheduling, achieving three-level cache collaboration. The database module is used for data persistence through database writes. The monitoring and maintenance module is used for metric collection and scaling. Each module is deployed independently and supports Kubernetes auto-scaling. Scaling is triggered when write QPS ≥ 8000 and write QPS ≤ 3000; and when CPU utilization ≥ 70% and CPU utilization ≤ 30%. This can improve resource utilization by 50%.

[0110] The system is divided into multiple independently deployed modules, supporting fine-grained scaling. Through the close integration of module division with core logic such as caching, writing, and compensation—for example, the independent cache scheduling module—complex three-level cache coordination strategies can be upgraded and scaled independently without affecting the main data processing chain. This modular design maximizes the system's performance.

[0111] By applying the technical solutions of the above embodiments, the high-speed information interaction system based on the three-level cache coordination mechanism described in the above embodiments first obtains the target high-speed information when writing data, and sends the target high-speed information to the write topic of the Kafka message queue, so as to allocate information partitions in the write topic according to the hash ratio score. Then, the consumer group of the microservice node pulls the Kafka message queue and generates a unified cache key. Then, based on the unified cache key, the target high-speed information is written to the local second-level cache, the distributed third-level cache, and the database in parallel through the microservice thread pool. The system can be based on a layered architecture of middle platform first-level cache, microservice local second-level cache, and distributed third-level cache, and uses a thread pool to update high-speed information in parallel. Combined with the hash ratio score partitioning rule, it ensures the order of the same entity event and improves the throughput and efficiency of information interaction.

[0112] It should be noted that other corresponding descriptions of the functional units involved in the high-speed information interaction system based on a three-level cache collaborative mechanism provided in the embodiments of this application can be found in the corresponding descriptions in the high-speed information interaction method based on a three-level cache collaborative mechanism provided in the above embodiments, and will not be repeated here.

[0113] This application also provides a computer device, specifically a personal computer, server, network device, etc. The computer device includes a bus, processor, memory, and communication interface, and may also include input / output interfaces and a display device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores location information. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0114] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0115] In one embodiment, a computer-readable storage medium is also provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0116] In one embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0118] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0119] Any references to memory, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0120] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0121] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.< / failedevent> < / boolean>

Claims

1. A high-speed information exchange method based on a three-level cache coordination mechanism, characterized in that, The method includes: Acquire target high-speed information, which includes business data generated from external data sources; The target high-speed information is sent to the write topic of the Kafka message queue through the expressive state transfer interface, so as to allocate information partitions in the write topic according to the hash ratio score; Consumer groups of microservice nodes pull the Kafka message queue and generate a unified cache key by parsing the target high-speed information; Based on the unified cache key, the three-level cache write is executed in parallel through a microservice thread pool to write the target high-speed information to the target writing medium, which includes a local second-level cache, a distributed third-level cache, and a database.

2. The method according to claim 1, characterized in that, The method further includes: Obtain information query requests, including user query requests input by the user through the application programming interface gateway; Using the application programming interface gateway, the unified cache key is generated based on the information query request; Multi-level cache queries are performed based on the unified cache key to retrieve the target high-speed information from the target query medium; the target query medium includes a central platform first-level cache, a local second-level cache, a distributed third-level cache, and a database; The target high-speed information is backfilled into the target query medium according to a preset cache order; the preset cache order is the order from the distributed level 3 cache, through the local level 2 cache to the middle platform level 1 cache.

3. The method according to claim 2, characterized in that, Performing multi-level cache queries based on the unified cache key to retrieve target high-speed information from the target query medium includes: A first query request is generated based on the unified cache key; In response to the first query request, the target high-speed information is queried from the first-level cache of the middle platform; If a match is found in the first-level cache of the middle platform, the target high-speed information returned by the first-level cache of the middle platform is extracted; If the query fails to find the target high-speed information in the first-level cache of the central platform, the target high-speed information is queried in the local second-level cache according to the unified cache key.

4. The method according to claim 3, characterized in that, Querying the target high-speed information in the local secondary cache according to the unified cache key includes: A second query request is generated based on the unified cache key; In response to the second query request, the target high-speed information is queried from the local second-level cache; If a query hits in the local second-level cache, the target high-speed information returned by the local second-level cache is extracted, and the target high-speed information is used to synchronously update the middleware first-level cache; If the query in the local second-level cache is not found, the target high-speed information is queried in the distributed third-level cache according to the unified cache key.

5. The method according to claim 4, characterized in that, Querying the target high-speed information in the distributed three-level cache according to the unified cache key includes: A third query request is generated based on the unified cache key; In response to the third query request, the target high-speed information is queried from the distributed three-level cache; If a query hits in the distributed three-level cache, the target high-speed information returned by the distributed three-level cache is extracted, and the target high-speed information is used to synchronously update the middle platform first-level cache and the local second-level cache; If the query in the distributed three-level cache is not found, the target high-speed information is queried in the database according to the unified cache key.

6. The method according to claim 5, characterized in that, According to the unified cache key, the target high-speed information is queried in the database, including: A fourth query request is generated based on the unified cache key; The fourth query request is sent to the database to query the target high-speed information in the database; If the query in the database is successful, the target high-speed information returned by the database is extracted, and the target high-speed information is used to synchronously update the middle platform first-level cache, the local second-level cache, and the distributed third-level cache. If the query fails to find a match in the database, a query failure message will be generated.

7. The method according to claim 1, characterized in that, The method further includes: Monitor the exception classes during the data writing process; When the detected exception class is detected, a cache rollback is performed to restore the local second-level cache and the distributed third-level cache to the state before the target high-speed information was written. Based on the detected anomaly type, query the failed write event, and the failed write event is associated with the target high-speed information; The failure is written to the target high-speed information associated with the event and delivered to the retry topic of the Kafka message queue; Based on the retry topic, the target high-speed information is rewritten to the L3 cache.

8. The method according to claim 7, characterized in that, After re-performing the L3 cache write to the target high-speed information based on the retry topic, the method further includes: Record the number of failed retries for re-execution of the L3 cache write; When the number of failed retries reaches a preset retry threshold, the failure event is written to a persistent log, which is stored in a log data analysis platform. The persistent log is scanned based on a timed compensation task to identify the failed write event; A three-level cache write is performed on the target high-speed information associated with the failed write event.

9. The method according to claim 1, characterized in that, The method further includes: Calculate the data popularity of the target highway information, whereby the data popularity includes the number of queries to the target highway information per unit time. The popularity threshold is obtained, which includes a first popularity threshold set for the local second-level cache and a second popularity threshold set for the distributed third-level cache; the first popularity threshold is greater than the second popularity threshold. Based on the data popularity, the target high-speed information is synchronized to the cache space associated with the popularity threshold.

10. A high-speed information interaction system based on a three-level cache coordination mechanism, characterized in that, The system includes: An external real-time data source module is used to acquire target high-speed information, which includes business data generated by an external data source. The Kafka message queue module is used to send the target high-speed information to the write topic of the Kafka message queue through the expressive state delivery interface, so as to allocate information partitions in the write topic according to the hash ratio score; The microservice node cluster module is used to pull the Kafka message queue using the consumer group of the microservice nodes, and generate a unified cache key by parsing the target high-speed information; The information writing module is used to write the target high-speed information to the target writing medium by performing three-level cache writing in parallel through a microservice thread pool based on the unified cache key. The target writing medium includes a local second-level cache, a distributed third-level cache, and a database.

Images