Distributed data preloading and querying method and device and storage medium

Abstract

The invention discloses a distributed data preloading and querying method and device and a storage medium. The method comprises the steps of searching data corresponding to a query key in a cache region integrated by an application server; if the query result of the query key is hit in the cache region, returning the query result; if the query result of the query key is not hit in the cache region, determining whether a target query key which is the same as the query key exists in the executed database query; and if the target query key does not exist, initiating a query request based on the query key to the database, and storing a query result returned by the database into a cache region integrated by the application server to respond to the query of the same query key. According to the method and the device, the cache is integrated in the application server, the resource overhead of an independent cache server is eliminated, the concurrent query when the cache is missed is merged and processed, and the concurrency is changed into single query, so that the repeated request pressure of the database is reduced fundamentally, and the risk of cache breakdown is effectively prevented.

Description

Technical Field

[0001] This application relates to the field of distributed system data management technology, and in particular to a distributed data preloading and querying method, device and storage medium. Background Technology

[0002] In distributed systems such as large-scale multiplayer online games and social platforms that require processing massive amounts of real-time interactive data, the business logic server frequently queries the central database to obtain critical data such as user status and configuration information. This is one of the most critical and load-intensive operations. To ensure the system's response speed and stability and to cope with high concurrency access pressure, the design of the data query architecture is crucial.

[0003] Currently, a common solution is to allow business servers to directly access relational databases (such as MySQL). This model is simple to implement, but in actual operation, especially when facing a surge in requests, a large number of queries with the same or similar content can directly overwhelm the database. Since each query requires an independent connection establishment, SQL parsing, execution, and result return, the database's CPU, memory, and I / O resources quickly become bottlenecks, leading to increased response latency and potentially even service unavailability. Its fundamental flaw lies in the lack of a merging and buffering mechanism for concurrent requests, failing to transform repetitive database accesses into single operations, thus subjecting the database to unnecessary and enormous pressure.

[0004] To address the aforementioned database pressure issues, introducing a dedicated distributed caching middleware (such as Redis) has become a standard architecture. In this model, the business server first queries the dedicated caching cluster, accessing the database only when the cache misses. While this solution reduces the direct load on the database, it also introduces significant new challenges. First, deploying and maintaining a highly available dedicated caching cluster requires dedicated server hardware, network configuration, and continuous operational investment. Furthermore, in many business scenarios, access load exhibits distinct peaks and troughs; during low-load periods, the resource utilization of dedicated caching servers is often very low, resulting in wasted hardware investment and energy. Second, the complexity of the system architecture and maintenance costs increase significantly. Business logic changes from directly accessing the database to interacting sequentially with the caching layer and the database layer, increasing the number of links and potential points of failure. This not only increases network latency and overall system complexity but also makes deployment, monitoring, and fault diagnosis more difficult. More importantly, this architecture harbors the risk of cache invalidation. For example, when some hot data expires or is cleared in the cache, a large number of concurrent requests will "penetrate" the cache layer and directly access the database. This "cache breakdown" phenomenon can easily cause the backend database to be overloaded instantly, which may trigger a chain reaction and cause a system-wide avalanche.

[0005] It is evident that in the traditional mainstream technology path, the solution of directly accessing the database cannot cope with high concurrency, while the solution of introducing independent caching, while alleviating the pressure on the database, comes at a high cost in terms of resource efficiency, architectural complexity, and system risk.

[0006] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0007] The main purpose of this application is to provide a distributed data preloading and querying method, device and storage medium, which aims to solve the technical problems of resource waste, complex architecture and cache breakdown risk of existing caching solutions.

[0008] To achieve the above objectives, this application proposes a distributed data preloading and querying method, the method comprising: Obtain historical query records and analyze the frequency and time distribution patterns of each query key based on the historical query records; Based on the frequency and time distribution pattern, one or more high-frequency query keys are identified from the historical query records; During a preset low-load period or after the application server starts, a preload query request based on the high-frequency query key is proactively sent to the database. Receive the preloaded query results returned by the database, and store the preloaded query results in advance in the cache area integrated by the application server; In response to a subsequent data query request, the data query request includes a query key; Based on the data query request, the cache area is searched to see if there is a query result that matches the query key; If the query result corresponding to the query key is found in the cache area, the query result is returned directly.

[0009] In one embodiment, after the step of checking whether a query result matching the query key exists in the cache area based on the data query request, the method further includes: If the query result corresponding to the query key is not found in the cache area, determine whether there is a target query key that is the same as the query key in the database query being executed; If the target query key does not exist, a real-time query request based on the query key is initiated to the database, and the real-time query result returned by the database is stored in the cache area integrated by the application server.

[0010] In one embodiment, after the step of determining whether a target query key identical to the query key exists in the database query being executed, the method further includes: If the target query key exists, then the current data query request is associated with the database query being executed corresponding to the target query key and registered accordingly. After the database query corresponding to the target query key is completed, the unified query result returned by the database will be distributed to all associated registered data query requests.

[0011] In one embodiment, after the step of determining whether a target query key identical to the query key exists in the database query being executed, the method further includes: If the target query key does not exist, create a new entry based on the current query key in the pre-maintained incomplete query registration table; A corresponding database query is created based on the new entry tag.

[0012] In one embodiment, prior to the step of responding to a subsequent data query request, the method further includes: Parse the target business data to be queried and extract the server identifier, business logic type identifier, and data entity identifier from the server cluster; Using predetermined encoding rules, the server identifier, business type identifier, and entity identifier are combined into a unique three-dimensional composite primary key, which serves as the query key.

[0013] In one embodiment, the step of storing the query result in the cache area includes: The query results are serialized and converted into compact structured data blocks; The structured data blocks are associated with the corresponding query keys and stored using a binary large object format, and their storage timestamps are recorded.

[0014] In one embodiment, after the steps of associating the structured data block with the corresponding query key and recording its storage timestamp, the method further includes: The cleanup process is triggered periodically or when the cache space reaches a threshold. Traverse the cache area, determine and clear expired query results based on the storage timestamp and predefined expiration policy; The expiration strategy configures different lifespan times for preloaded query results and real-time query results.

[0015] In one embodiment, the distributed data preloading and querying method further includes: Detect real-time load metrics of multiple candidate server nodes, including at least one of CPU utilization, memory utilization, and network I / O; According to the load balancing algorithm, the node with the lowest load is selected from the candidate server nodes as the application server to respond to the preloaded query request and the query request; The preloaded query request and the query request route are directed to the application server.

[0016] In addition, to achieve the above objectives, this application also proposes a distributed data preloading and querying device, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the distributed data preloading and querying method described above.

[0017] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the distributed data preloading and querying method described above.

[0018] One or more technical solutions proposed in this application have at least the following technical effects: The technical solution of this application involves acquiring historical query records and analyzing the frequency and time distribution patterns of each query key based on these records; identifying one or more high-frequency query keys from the historical query records based on the frequency and time distribution patterns; proactively initiating a pre-loading query request based on the high-frequency query keys to the database during a preset low-load period or after the application server starts; receiving the pre-loading query results returned by the database and storing the pre-loading query results in advance in a cache area integrated by the application server; responding to subsequent data query requests, which include query keys; searching the cache area for a query result matching the query key based on the data query request; and directly returning the query result if a query result corresponding to the query key is found in the cache area.

[0019] This application eliminates the resource overhead of a separate cache server by integrating a cache within the application server. It also reduces the pressure of repeated database requests by merging concurrent queries when the cache misses, turning them into single queries, and effectively prevents the risk of cache breakdown. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the first embodiment of the distributed data preloading and querying method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the distributed data preloading and querying method of the present invention; Figure 3 This is a flowchart illustrating the third embodiment of the distributed data preloading and querying method of the present invention; Figure 4 This is a flowchart illustrating the fourth embodiment of the distributed data preloading and querying method of the present invention; Figure 5 This is a flowchart illustrating the fifth embodiment of the distributed data preloading and querying method of the present invention; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the distributed data preloading and querying method in this application embodiment.

[0023] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] The main solution of this application embodiment is as follows: Obtain historical query records and analyze the frequency and time distribution patterns of each query key based on these records; identify one or more high-frequency query keys from the historical query records based on the frequency and time distribution patterns; proactively initiate a pre-loading query request based on the high-frequency query keys to the database during a preset low-load period or after the application server starts; receive the pre-loading query results returned by the database and store the pre-loading query results in advance in the cache area integrated by the application server; respond to subsequent data query requests, which include query keys; search the cache area for a query result matching the query key based on the data query request; if a query result corresponding to the query key is found in the cache area, the query result is directly returned.

[0027] In traditional mainstream technology paths, direct database access cannot handle high concurrency, while introducing independent caching can alleviate database pressure, but at a high cost in terms of resource efficiency, architectural complexity, and system risk.

[0028] This application provides a solution that reduces the pressure of repeated database requests by merging concurrent queries when the cache misses, turning them into single queries, and effectively preventing the risk of cache breakdown.

[0029] Based on this, embodiments of this application provide a distributed data preloading and querying method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the distributed data preloading and querying method of this application. In this embodiment, the distributed data preloading and querying method includes steps S10 to S70: Step S10: Obtain historical query records and analyze the frequency and time distribution pattern of each query key based on the historical query records; In this embodiment, before the application server begins processing data query requests, or as part of system initialization or periodic maintenance processes, a data query pattern collection and analysis process is performed. First, the application server continuously records each data query request initiated or attempted to be initiated into the database through its integrated monitoring and logging module. Each record related to a historical data query request includes at least a query key, a timestamp of the query initiation, and the data size information of the query result. The query key is a unique identifier pointing to a specific data entity in the database. In the specific implementation of this embodiment, the structure of the unique identifier is limited to a predefined triple form based on business logic, such as (server identifier, business type identifier, entity identifier).

[0030] After obtaining the historical query records, a background analysis process is initiated to process the set of historical query records offline or in near real-time. The core data processing logic of this analysis process lies in statistics and pattern recognition. Specifically, for the frequency analysis of the query key, the historical query records are traversed and aggregated using a hash table data structure. For each unique query key, the number of requests for that unique query key within a specified time window is accumulated. The time window is typically set to a period that reflects the business cycle, such as the most recent 24 hours or 7 days, and its specific length is pre-configured according to the business characteristics of the application server.

[0031] Simultaneously, analysis based on time distribution patterns is performed. The timestamp information is associated with the corresponding query keys and categorized according to evenly divided time slots (e.g., every hour, every half hour). By analyzing the changes in request volume for the same query key across different time slots, it is possible to identify whether the access pattern for the same query key is evenly distributed, exhibits periodic peaks (e.g., at specific times each day), or experiences random bursts. The key technical means for this analysis involve the grouping and statistical calculation of time series data.

[0032] In the specific implementation process, the analysis process generates and maintains a query key metadata table. This table records two core metrics for each tracked query key: the total number of requests within the analysis time window, i.e., the frequency of occurrence; and the request frequency distribution vector within each time slot, used to characterize the time distribution pattern. This process provides an accurate data foundation for subsequent identification of high-frequency and hotspot data, and all analyses are based on real request logs generated during actual system operation.

[0033] Step S20: Identify one or more high-frequency query keys from the historical query records based on the occurrence frequency and time distribution pattern; In this embodiment, the query key metadata table is generated and maintained, and the identification and filtering process of high-frequency query keys is executed. The core of the data processing logic of the identification and filtering process is to automatically filter out the data subset that has the greatest impact on system performance from a massive number of query key records, namely the high-frequency query keys, based on a preset, quantifiable threshold strategy.

[0034] First, the frequency of occurrence recorded in the query key metadata table is read. The basic and direct filtering strategy for this frequency of occurrence is to set an absolute frequency threshold. For example, the system is configured with a minimum request count threshold (e.g., 100 times per minute or 10,000 times per day). Any query key whose total request count exceeds the minimum request count threshold within the analysis time window will be initially marked as a candidate high-frequency query key. Second, to more accurately reflect the concentration of query load, a relative frequency threshold strategy can be used. This strategy calculates the distribution of the total request count for all query keys (e.g., calculating the P95 or P99 quantiles), and identifies the query keys with the highest request count in the top 5% or top 1% as high-frequency query keys.

[0035] Furthermore, more refined identification requires combining the aforementioned time distribution patterns. For example, a "hotspot time slot" can be defined as a period when the system's average load exceeds a preset level (e.g., CPU utilization exceeds 80%). For each query key, the proportion of its request volume within the hotspot time slot is analyzed. If the proportion of a query key's request volume within the hotspot time slot exceeds a specific threshold (e.g., 70%), even if the overall global frequency does not reach the absolute threshold, it is identified as a critical high-frequency query key because its load highly overlaps with the system's peak pressure, thus further enabling targeted measures to alleviate database pressure during peak periods.

[0036] Furthermore, the key technical means based on the high-frequency query keys involve data sorting, filtering, and threshold comparison algorithms. Through the algorithm shown, the background service performs an identification task periodically (e.g., hourly), that is, loading the latest query key metadata table from persistent storage, and sequentially applying one or more of the above-mentioned threshold strategies (frequency distribution index, time distribution pattern, and data sorting, filtering, and threshold comparison algorithms) to filter and generate the latest high-frequency query key list. The high-frequency query key list is stored in the memory of the application server and may be accompanied by a priority score (e.g., calculated by combining frequency and time urgency) for subsequent use.

[0037] In this embodiment, by setting multiple strategies for high-frequency query keys, it is ensured that the target of cache preheating is the data entities that are most likely to put pressure on the database and have the greatest impact on response speed.

[0038] Step S30: During a preset low-load period or after the application server starts, actively initiate a preload query request to the database based on the high-frequency query key. In this embodiment, after identifying the list of high-frequency query keys, an active cache preheating strategy is implemented. The core data processing logic of the cache preheating strategy is to load high-frequency data into the local cache in advance when the database is relatively idle or the system resources are sufficient, so as to avoid cache misses during peak business periods and impacting the database.

[0039] Based on this, the key technical means of the cache hot strategy lies in the determination of the "preset low-load period" and the scheduling of the preheating task triggering time. The determination of the preset low-load period is based on the monitoring of system performance indicators of the application server and database server cluster. That is, the system continuously collects multiple indicators including CPU utilization, memory utilization, database connection count, and network I / O traffic. By analyzing the above multiple indicators of historical monitoring data, typical load patterns can be learned. For example, the overall cluster load is usually below 20% from 2:00 AM to 5:00 AM every day, which is determined as a low-load period. Alternatively, the administrator can manually configure one or more specific time windows as low-load periods.

[0040] The preheating task triggering of the above-mentioned cache hot strategy is set in two main modes. The first is time-based scheduling triggering: the system's built-in task scheduler (such as Cron Job) automatically starts the preheating process at the beginning of the configured low-load period. The second is triggering based on relevant control events: after the application server instance completes a cold start or restart, and before starting formal service business requests, a preheating process is immediately initiated. In addition, real-time load judgment can be combined; when the monitoring system detects that the current cluster load is lower than the dynamically calculated safety threshold, an incremental preheating is triggered.

[0041] In the specific implementation process, when the preheating conditions described above are met, the high-frequency query key list is read from memory. For each query key in the high-frequency query key list, the application server constructs a standard database query request. The format of the database query request is exactly the same as that of a normal business query, ensuring the consistency of the query results. Then, through its database access layer, a series of preloading query requests are proactively initiated to the database in a batch or serial manner. To improve efficiency, requests corresponding to multiple query keys can be packaged into a single batch query (if the database supports this), or a connection pool can be used to maintain persistent connections to reduce the overhead of establishing connections. This allows the loading of critical data to be completed silently before peak business periods.

[0042] Step S40: Receive the preloaded query results returned by the database, and store the preloaded query results in advance in the cache area integrated by the application server; In this embodiment, after the preloaded query request is received and processed by the database, the database returns the corresponding query result dataset. The core data processing logic of this query result dataset return is to receive the query result dataset and persistently store the query result dataset in the local cache area of ​​the application server in a format compatible with business queries, thereby completing the filling of cache data.

[0043] Furthermore, after receiving the preloaded query results returned by the database, the database access layer based on the application server first verifies the query result dataset, such as checking data integrity and whether the format meets expectations. After successful verification, it enters the cache storage stage. The cache area is physically a managed data structure allocated in the memory of the application server process, typically implemented as a memory key-value store based on a hash table or similar efficient retrieval structure.

[0044] Furthermore, the key technical means for caching the query result dataset involves the structured encapsulation and storage of cache entries. Specifically, each cache entry uses the high-frequency query key (i.e., the triple) as a unique key. The corresponding value is not the original database row data, but rather a structured data object. This data object contains at least two parts: first, the valid data payload of the preloaded query results returned from the database; and second, the associated metadata, such as the data version number, the timestamp of loading into the cache, and the preset expiration time (TTL). Encapsulating the data payload and metadata together facilitates subsequent cache consistency management and invalidation policy implementation.

[0045] The specific implementation process of the above caching operation is as follows: For each returned query result, a cache key is generated based on its corresponding original query key. Then, the encapsulated data object is stored in the hash table of the cache area.

[0046] Furthermore, if the cache area employs a sharded or partitioned design, its specific shard location needs to be determined based on the cache key using algorithms such as consistent hashing. Storage operations require atomicity or thread-safe mechanisms to prevent data inconsistency in rare concurrent warm-up scenarios. Simultaneously, the system updates its internal cache directory or index to record the currently pre-loaded key set and its metadata, facilitating rapid determination of whether a frequently queried key might exist in the warm-up cache. At this point, the data corresponding to the frequently queried key has been transformed from the result of a passive database query into a cache resource that can be quickly accessed locally on the application server.

[0047] Step S50: In response to a subsequent data query request, the data query request includes a query key; In this embodiment, when the application server is in normal operation and providing data query services, it can receive and process external data requests. Based on the entry logic for receiving and processing these data requests, the data query request originates from a business logic server communicating with the application server, such as a game server (GS) or a session control server (CCS). The external data request aims to obtain business data of a specific entity to support core functions such as game logic operation and user status display.

[0048] Furthermore, the format and protocol of the data query request need to be predefined to ensure that both communicating parties can correctly parse it. In the specific implementation of this embodiment, the data query request is encapsulated as a Remote Procedure Call (RPC) message. The RPC message contains at least one required parameter, namely the query key. The structure of the query key must be strictly consistent with the format defined and identified internally by the system, such as the (server identifier, business type identifier, entity identifier) ​​triple form mentioned above, thus serving as the basis for all subsequent cache lookup and database query operations.

[0049] In practice, the application server opens one or more network ports to listen for RPC connections from authorized clients. Upon receiving a complete RPC request data packet, the application server's network communication module first decodes and verifies it to check the legitimacy of the request source, the integrity of the message, and the correctness of the format based on the RPC request data packet.

[0050] After successful verification, the RPC method name and parameter list of the RPC request data packet are parsed out. Based on the parsing result, after determining that the RPC request data packet is a method call identified as a data query, the query key is extracted from the parameter list. The extracted query key is encapsulated into an internal request context object. In addition to the query key itself, the internal request context object also includes information such as the timestamp of the request arrival, the request sequence number, and the source client identifier, which are used for subsequent request tracing, logging, and distinguishing different request sources in advanced features such as request merging.

[0051] Subsequently, the encapsulated internal request context is submitted to the core request processing pipeline of the application server. At this point, an external data acquisition request has been transformed into a clear data retrieval task within the system. The goal of this transformation is to locate the corresponding data value using the query key, preparing it for the next stage of cache retrieval or database query. Because this process requires high concurrency, asynchronous non-blocking I / O and an event-driven architecture are typically used to implement this transformation.

[0052] Step S60: Based on the data query request, search the cache area to see if there is a query result that matches the query key; In this embodiment, after receiving and encapsulating the internal request context containing the query key, it immediately attempts to retrieve data from the local cache resource. The operation of retrieving data from this internal cache resource is a key step in reducing database load and improving response speed. The core data processing logic of this data retrieval is to perform a fast key-value lookup operation to determine whether the required data already exists in the memory of the application server.

[0053] Specifically, the cache region is typically organized internally as one or more concurrent-safe, high-performance key-value mapping data structures, such as using a language-native hash table (which might be `table` in Lua) in conjunction with read-write locks or more efficient lock-free data structures. The lookup process based on the cache region uses the query key as input. Since the query key is a structured composite key (such as a triple), it first needs to be converted into a single representation suitable as a hash table key. Based on this conversion, the three integer components of the triple are concatenated or hashed using a serialization function (e.g., using `tostring(serverId)..""..tostring(bizTypeId)..""..tostring(entityId)` or a more efficient hash algorithm) into a unique string or integer hash value, which serves as the actual key for lookup in the cache hash table.

[0054] In practice, the request processing thread retrieves the query key from the internal request context and calls a unified key generation function to obtain the cache lookup key. Subsequently, the request processing thread attempts to access the data structure of the cache area and performs a lookup operation, which is a memory access and extremely fast. The lookup result has two possibilities: a hit or a miss. Specifically, a hit means that an entry with the lookup key as its index exists in the cache hash table. The system only needs to confirm the existence of the entry; it also needs to further check its validity. This involves reading the metadata associated with the cache entry, especially the expiration time (TTL), and comparing it with the current system time. If the entry has not expired, it is considered a valid hit; if it has expired, although it physically exists, it should logically be considered a miss, and asynchronous cache cleanup or update logic may be triggered.

[0055] The lookup process is designed to be thread-safe, ensuring that under high-concurrency queries, it will not crash or return dirty data due to concurrent modifications to the data structure. Regardless of whether a match is found or not, the lookup operation will generate a clear status flag (such as a boolean value `isCacheHit`) and a cached data object if a match is found. Both the status flag and the status result are passed to the request context, determining whether the subsequent process should directly return the data or continue querying the database.

[0056] Step S70: If the query result corresponding to the query key is found in the cache area, the query result is returned directly.

[0057] In this embodiment, when a query result corresponding to the query key is found in the cache area, it indicates that a valid query result matching the query key has been successfully found in the cache area. The query result is then returned. The return of the query result can also be defined as terminating the further query link to the database and directly using the cached data to complete the response of this request, thereby achieving the specific operation of returning data at the millisecond level.

[0058] At this point, the internal request context already holds the complete data object retrieved from the cache area. This complete data object contains the original data payload and related metadata. The primary operation of this return process is to extract the business data portion, i.e., the query result, from the complete data object. The extraction process can be defined as direct field access because the complete data object is structurally encapsulated when stored in the cache area, thus enabling extraction through direct field access. The extracted query result is converted into a return format that conforms to the data query request. For example, if the original request expects to return a Protobuf or JSON format data packet, the corresponding serializer needs to be called to serialize the in-memory data structure (such as a table or object) into a byte stream or string.

[0059] The key technical means involved in the operations based on the above query results are response construction and efficient return. Specifically, while preparing the response data, specific monitoring indicators can be updated, such as incrementing the cache hit rate counter as a monitoring indicator condition for response construction and efficient return.

[0060] Subsequently, the serialized response data payload, along with the required response header information (such as the status code "success", data version number, etc.), is encapsulated into a standard RPC response message. The RPC response message is then sent back to the client that initiated the data query request via the network communication module of the application server, following the original path (i.e., the network connection that received the request).

[0061] In the specific implementation, the return operation emphasizes high efficiency and low latency, avoiding any unnecessary blocking operations such as disk I / O or additional network communication throughout the process. Therefore, it is necessary to ensure that data always flows in memory. In an asynchronous architecture, when a cache hit occurs, the processing thread or coroutine can immediately process the response without waiting for database I / O events. This significantly reduces the end-to-end latency of requests. Furthermore, by implementing a request merging mechanism, other concurrent requests with the same query key that are waiting to be merged when a cache hit occurs can also be responded to immediately with this cached result, further improving overall throughput. Thus, this embodiment completes the high-performance cache response process without subjecting the database to any query pressure, thereby further reducing the pressure of repeated database requests and effectively preventing the risk of cache breakdown.

[0062] Furthermore, you can also view Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the distributed data preloading and querying method of the present invention, based on the shown... Figure 2 After the step of searching the cache area for a query result matching the query key based on the data query request, the method further includes steps S80-110: Step S80: If the query result corresponding to the query key is not found in the cache area, determine whether there is a target query key that is the same as the query key in the database query being executed; Step S90: If the target query key does not exist, initiate a real-time query request based on the query key to the database, and store the real-time query result returned by the database into the cache area integrated by the application server.

[0063] Step S100: If the target query key exists, then associate and register the current data query request with the database query being executed corresponding to the target query key. Step S110: After the database query corresponding to the target query key is completed, the unified query result returned by the database is distributed to all associated registered data query requests.

[0064] In this embodiment, when it is determined that no query result matching the query key is found in the cache area, a cache miss processing flow is initiated. The core data processing logic based on this flow lies in preventing repeated impacts on the database at the moment of cache invalidation by introducing concurrency control and request merging mechanisms. Specifically, it is necessary to determine whether there is currently an ongoing database query for the exact same query key, that is, to determine whether there is a target query key that is the same as the query key currently being requested.

[0065] The above confirmation operation relies on a globally shared, concurrently safe data structure, which can also be represented as an "in-process query mapping table" or an "incomplete query registration table". This data structure maintains the status of all queries that have been initiated but have not yet returned results from the database. In this embodiment, the data structure is defined as an in-process query mapping table. In the incomplete query registration table, the registered key is the query key that is being queried, and its value is defined as an object that identifies the status or a list of requests waiting for callback.

[0066] When a cache miss occurs, the processing thread first attempts to perform a lookup in the in-process query mapping table using the currently defined query key as an index. The lookup operation is defined as atomic or lock-protected to ensure consistency of state judgment in a high-concurrency environment. If the lookup succeeds, meaning an entry with the exact same query key exists in the in-process query mapping table, it indicates that a database query operation for that target key already exists and is in progress. If the lookup fails, it indicates that there is currently no ongoing query for the same key.

[0067] If the target query key is not found in the in-execution query mapping table, a new database query needs to be initiated. First, a real-time query request based on the current query key is sent to the database; the construction of this real-time query request is no different from a regular query. While waiting for the database response, as a key technical measure, the processing thread performs a pre-processing action: registering the query key and the context information of the query initiation in the in-execution query mapping table in an atomic operation, marking the query for this key as "in progress." This operation aims to provide a basis for merging subsequent concurrent requests for the same key. After the database returns the real-time query result, the system performs two key operations: first, storing the query result and its metadata in the cache area to complete cache filling; second, removing the entry corresponding to the query key from the in-execution query mapping table, marking the end of this query cycle. Finally, the query result is returned to the data query request.

[0068] Execution is performed upon determining the existence of the target query key. Request merging is also required. The core of request merging lies in "suspending" or "registering" the current data query request to an existing query task, rather than initiating a duplicate query. In specific implementation, the processing thread accesses the value structure corresponding to the target query key in the query mapping table. This value structure is set as a dynamic container, such as a first-in-first-out queue or a list, to store the request context of all requests waiting for the query result. The context information of the current request (such as callback functions, client connection handles, etc.) is safely appended to this container. After this "association registration" operation is completed, the processing thread or coroutine of the current data query request enters a waiting state, or its callback is temporarily stored, thereby releasing system resources to process other requests and avoiding blocking.

[0069] After the database query corresponding to the target query key is executed, a data distribution mechanism is triggered. That is, when the database returns the query result, the thread that initially initiated the query must not only store the result in the cache and clean up the mapping table entries in the callback function, but also undertake the responsibility of result distribution. The key technical means for this result distribution responsibility lies in traversing all associated request contexts registered in the value container of the corresponding value in the mapping table for the target query key. For each registered item, a response conforming to the expected format of the data query request is constructed using the unified query result obtained from the database, and sent one by one through their respective saved response paths (such as RPC callbacks). This data distribution process ensures that all concurrent requests for the same data arriving during the initial query execution ultimately receive the same query result, while the database only bears the query pressure once, fundamentally eliminating the risk of database avalanche caused by cache breakdown.

[0070] You can also view Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the distributed data preloading and querying method of the present invention, based on the shown... Figure 3 After the step of determining whether a target query key identical to the query key exists in the database query being executed, the process further includes steps S120-130: Step S120: If the target query key does not exist, create a new entry based on the current query key in the pre-maintained incomplete query registration table; Step S130: Create a corresponding database query in execution based on the new entry tag.

[0071] In this embodiment, during the internal operation process under the branch where the target query key does not exist, the execution status of the new database query needs to be initialized in a standardized manner. The lifecycle of the new database query based on the target query key is marked and tracked through a centralized registration table to ensure that the same query arriving within a very short time window can be correctly identified and merged.

[0072] This operation is performed immediately after determining that the target query key is not found in the query mapping table during execution. That is, it is a critical preparation phase before starting a new query, announcing the start of the query cycle for the current query key in an atomic manner.

[0073] In specific implementation, the pre-maintained incomplete query registration table is the in-process query mapping table defined in the above embodiments. The in-process query mapping table is defined as a globally visible concurrent data structure within the application server process. To create a new entry, a unique index key within the in-process query mapping table needs to be generated for the current query key. This is typically achieved by performing a hash operation on the query key (such as a triple) (e.g., using algorithms like MD5 or CityHash to generate an integer or string hash value), with the hash value serving as the primary key of the registration table.

[0074] The process of creating a new entry is defined as an atomic check-create operation, preventing multiple threads from simultaneously determining that the entry does not exist and creating duplicate entries under extremely high concurrency. The key technical means for this creation operation lies in using a concurrent hash table that supports atomic operations, or protecting the modification area of ​​the query mapping table during execution using a mutex. The atomic operation ensures that only one thread successfully creates the new entry identified by the hash value of the current query key. The value structure of the new entry contains at least two core pieces of information: first, a query status flag, initialized to "querying"; and second, a set of request contexts for storing waiting results, initially empty, or possibly containing the request context of the currently initiating query. Once the new entry is successfully created, it means that the current query key is officially placed in a "querying" state, and any subsequent query requests for the new entry will be merged based on this entry.

[0075] Based on the newly created entry, a database query operation is initiated. In the processing thread for this query operation, firstly, the thread needs to extract the complete query key from the current request context and construct a specific database query statement or call parameters based on the query key. Then, an asynchronous query request is initiated to the database through the application server's database access proxy (such as `dba_agent`). A key associated operation when initiating the asynchronous query request is binding the future callback function of the asynchronous query request to the entry in the registry. This constraint clarifies that when calling the database query interface, the callback function passed must be able to access the unique identifier of this entry (such as a hash key or memory reference) so that the entry requiring updating and notification can be accurately located upon completion of the query.

[0076] In the specific implementation, the database query based on the new entry is initiated asynchronously. That is, the thread initiating the query does not need to wait after submitting the request and can continue processing other tasks. The actual execution of this query is completed in the background by the database and its driver. Based on this, a complete and protected query initialization process is achieved, meaning the query status has been recorded in the central registry table, the asynchronous query task has been submitted, and a clear merging anchor point is provided for all subsequent requests for this key. The above mechanism ensures a critical transition period from "determining no duplicate queries" to "actually initiating the query," preventing state races.

[0077] You can also view Figure 4 , Figure 4 This is a flowchart illustrating the fourth embodiment of the distributed data preloading and querying method of the present invention. Figure 4 Before the steps that respond to subsequent data query requests, steps S140-150 are also included: Step S140: Parse the target business data to be queried and extract the server identifier, business logic type identifier, and data entity identifier from the server cluster. Step S150: Using predetermined encoding rules, the server identifier, business type identifier, and entity identifier are combined into a unique three-dimensional composite primary key, which serves as the query key.

[0078] In this embodiment, the stage after the data query request is received by the application server and before the formal cache query or database query process begins is essentially the data preprocessing and key construction stage. This data logic involves standardizing and uniquely transforming the diverse data acquisition intentions from the business layer into query identifiers that the system can efficiently process and identify—that is, the query key. This is the fundamental prerequisite for ensuring the correct operation of all subsequent mechanisms such as cache lookup and request merging.

[0079] The application server parses the received data query request, which is represented at the business layer as a call mechanism containing multiple parameters. The key technical approach lies in extracting specific, globally uniquely identifying dimension information from the structured request parameter set. Specifically, this dimension information is defined as: server identifier in the server cluster, business logic type identifier, and data entity identifier. Specifically, the server identifier distinguishes the physical or logical server partition where the data resides; the business type identifier distinguishes different data tables or business modules, such as user basic information, item configuration tables, and legion data; and the entity identifier uniquely identifies a specific data record within a specific business type and server partition, such as a user's role ID or an item's item ID.

[0080] During the parsing of the target business data, predefined communication protocols or API interface specifications can be followed. For example, if the request for the target business data is a Protobuf message transmitted via RPC, the corresponding deserialization code is invoked to read the values ​​of the three identifiers defined above from a specifically named field. The extraction operation is robust, including checks on the existence and type validity of parameters. The three extracted identifiers are temporarily stored in the request context object in memory as raw materials.

[0081] The extracted independent identifiers are combined into a single query key that can serve as an efficient retrieval key. This query key is formed by applying predetermined, deterministic encoding rules. Specifically, these rules ensure that for any two different sets of three-dimensional identifier inputs, the output result (i.e., the query key) is different, thus satisfying the requirement of uniqueness. In practical applications, these encoding rules are defined as string concatenation. For example, after converting the server identifier, business type identifier, and entity identifier into strings, they are connected with a specific delimiter (such as an underscore "_") to form a string key like "1001_5_40001".

[0082] Alternatively, the query key can be generated using mathematical combinations or hash functions to create a compact integer key. For example, bitwise operations can be used for combination: assuming each identifier is within a predetermined bit range (e.g., server ID occupies 16 bits, business type ID occupies 8 bits, entity ID occupies 32 bits), then the identifiers can be combined into a 64-bit long integer through shift and OR operations: `key = (serverId<<40) | (bizTypeId<<32) | entityId`. This constraint results in a fixed key length, fast comparison speed, and extremely high efficiency as a hash table key.

[0083] Alternatively, a consistent hash function (such as MurmurHash3) can be used to calculate the hash value of the concatenated string, which can then be used as the query key.

[0084] In practice, the specific algorithm for the encoding rules is loaded or built-in during initialization. When processing each request, a unified key generation function is called, passing in three identifier parameters. The function returns the calculated three-dimensional composite primary key. The generated primary key is set back to the request context object as the standard query key used in all subsequent processes.

[0085] Furthermore, based on the steps described in the first and second embodiments above, storing the query result in the cache area includes: The query results are serialized and converted into compact structured data blocks; The structured data blocks are associated with the corresponding query keys and stored using a binary large object format, and their storage timestamps are recorded.

[0086] In this embodiment, after the application server obtains the data to be stored in the cache, the query results, whether obtained through a preloading process or a real-time query process, must undergo standardized storage processing before being persisted in the cache area. The core data processing logic of this storage operation lies in converting and optimizing the format of the original, complexly nested query results, aiming to improve storage efficiency, reduce memory usage, and ensure that the data organization in the cache is consistent and manageable.

[0087] In practice, the query results need to be serialized. Serialization involves converting the data structure or object state of the cached area into a format that can be continuously stored or transmitted over a network. The query results are typically represented by native data structures of the programming language; for example, in a Lua server, it might be represented as a `table` containing multiple key-value pairs, where values ​​can be numbers, strings, booleans, or even nested `tables`. Therefore, directly storing this native structure would not only consume significant memory and metadata overhead but could also involve complex deep copies when shared across threads or processes. Thus, a predefined serialization function is called to convert the query results.

[0088] The key technical approach to this transformation lies in employing a compact, non-redundant serialization protocol. For example, MessagePack or a custom binary protocol can be used. This binary protocol iterates through each field of the query result, encoding the field name (key) and field value (value) into a compact binary representation. This eliminates the need for metadata such as type identifiers and hash table overhead in the original data structure, ultimately generating a continuous sequence of bytes—the compact structured data block. This structured data block not only contains all the information necessary to recover the original data but also represents a smaller data volume.

[0089] After serialization, the storage association stage begins. The structured data block is bound and stored with the query key that identifies it. In this embodiment, a binary large object format is explicitly specified for storage. In the core data structure of the cache area (such as a hash table), the query key (or its hash value) serves as the index key, and the corresponding value is a simple, opaque binary data block reference or container. This container contains at least two parts: first, a pointer to and length of a contiguous memory region storing the structured data block; and second, the associated metadata, which includes the storage timestamp. The storage timestamp records the system time when this entry was created or most recently updated (typically a Unix timestamp, accurate to milliseconds or microseconds), serving as the basis for subsequent cache invalidation and cleanup strategies.

[0090] In the specific implementation process, when it is determined that a cache entry needs to be stored, the data payload is serialized to generate a binary block. Subsequently, a contiguous block of memory of the same size is allocated in the cache manager, and the binary block is copied to the contiguous memory. Finally, using the query key as an index, an entry is created or updated in the cache hash table. The value of the entry points to the newly allocated memory block, and the current system time is synchronously recorded as a storage timestamp. This process ensures that cached data exists in a highly efficient, low-overhead binary form, facilitating fast reading and writing, while also providing necessary management information.

[0091] In addition to the steps of associating the structured data block with the corresponding query key and recording its storage timestamp, the method further includes: The cleanup process is triggered periodically or when the cache space reaches a threshold. Traverse the cache area, determine and clear expired query results based on the storage timestamp and predefined expiration policy; The expiration strategy configures different lifespan times for preloaded query results and real-time query results.

[0092] In this embodiment, in order to ensure that the memory resources of the cache area are effectively utilized and to maintain a certain degree of freshness of the cached data relative to the source data of the database, an active cache entry cleanup mechanism needs to be implemented. The core data processing logic of the cache entry cleanup mechanism is to identify and remove expired or potentially outdated cached data through periodic or event-driven scanning and evaluation, thereby freeing up memory space and making room for new data.

[0093] This cleanup process is triggered based on two main conditions; the cleanup task starts when either condition is met. The first is periodic triggering, which uses a background timer to issue cleanup events at preset fixed intervals (e.g., every 5 minutes or hour). The second is resource threshold-based triggering. This involves continuously monitoring the total memory usage or the number of stored entries in the cache area. When usage exceeds a preset threshold (e.g., reaching 80% of the maximum memory allocated to the cache, or the number of entries exceeding 1 million), emergency cleanup is immediately triggered, ensuring effective cache management during both routine maintenance and resource-constrained periods.

[0094] Once the cleanup process is triggered, the traversal and judgment process begins. Since caches are typically implemented as hash tables, directly traversing all buckets is feasible but may have a slight performance impact. In practice, the cleanup task is usually executed by a low-priority background thread to minimize the impact on frontend business request processing. During the traversal, for each cache entry, its associated metadata is read, particularly the storage timestamp. By obtaining the current time, the entry's lifetime (current time - storage timestamp) is calculated. Then, this lifetime is compared with a predefined expiration policy to determine whether the entry should be evicted.

[0095] The aforementioned expiration strategy is the core rule set of the cleanup logic. This key innovation lies in the ability to configure different lifespans for data from different sources. Specifically, independent lifespan configuration items, such as `TTL_PRELOAD` and `TTL_REAL_TIME`, are defined for query results stored through a pre-loading process and query results stored through a real-time query process. The source of the entry needs to be known during the determination process. This can be achieved by additionally recording a "data source" flag in the metadata during storage. For example, if `TTL_PRELOAD` is set to 7200 seconds (2 hours) and `TTL_REAL_TIME` is set to 300 seconds (5 minutes), then entries from pre-loaded data will be determined to expire if their lifespan exceeds 7200 seconds; entries from real-time queries will expire if their lifespan exceeds 300 seconds. This differentiated strategy aligns with business logic. Specifically, pre-loaded data is relatively static or has a low change frequency, allowing for a longer cache time; while real-time query data may be updated more frequently, requiring a shorter cache period to ensure stronger consistency.

[0096] During the cleanup operation, for entries deemed expired, their index keys are removed from the cache hash table, and the association is broken. Then, the memory space occupied by the expired entries and used to store the structured data blocks is released. Finally, optionally, the cache usage statistics are updated. The entire cleanup process continues in the background until all entries have been traversed or enough memory has been released so that the usage rate falls below a safe threshold. Through this fine-grained, configurable expiration management, a dynamic balance is achieved between cache hit rate and data timeliness, ensuring the sustainable utilization of cache resources.

[0097] Furthermore, you can also view Figure 5 , Figure 5 This is a flowchart illustrating the fifth embodiment of the distributed data preloading and querying method of the present invention, based on the shown... Figure 5 The distributed data preloading and querying method further includes steps S160-180: Step S160: Detect the real-time load metrics of multiple candidate server nodes, wherein the load metrics include at least one of CPU utilization, memory utilization, and network I / O. Step S170: According to the load balancing algorithm, select the node with the lowest load from the candidate server nodes as the application server to respond to the preloaded query request and the query request. Step S180: The preloaded query request and the query request route are directed to the application server.

[0098] In this embodiment, a dynamic, real-time load-based node selection mechanism is introduced when determining which specific server instance should be responsible for cache integration and query services. The core data processing logic of the node selection mechanism is to continuously evaluate the real-time status of available resources in the system and, based on the load balancing principle, intelligently schedule data processing tasks to the most suitable server node, thereby achieving optimized utilization of resources within the cluster and balanced distribution of the overall load.

[0099] Specifically, this operation focuses on the systematic collection and aggregation of real-time load status of multiple candidate server nodes. These candidate server nodes refer to all physical or virtual machine instances in the cluster that have deployed and are running the application server logic (i.e., the IMS service with integrated caching functionality). A lightweight monitoring agent is deployed on each candidate server node. This agent collects local system performance metrics at fixed time intervals (e.g., per second). These load metrics are explicitly defined as including at least one of CPU utilization, memory usage, and network I / O throughput. Specifically, CPU utilization is obtained by reading the operating system's ` / proc / stat` or calling system APIs; memory usage is obtained by calculating the percentage of used memory to total memory; and network I / O is measured by counting the number of bytes sent and received by a specified network interface card per unit time. The collected raw metric data is sent to a centralized monitoring service or stored in a shared storage (such as Redis or ZooKeeper) accessible to all nodes, forming a global, time-updated snapshot of the node load status. The data structure of the node load status snapshot uses the node ID as the key and the value as a tuple containing the values ​​of various metrics and the latest collection timestamp.

[0100] Based on the collected load status snapshots, a load balancing decision is executed. The core of this load balancing decision lies in dynamically selecting an optimal node from the candidate server nodes according to a predefined load balancing algorithm. The specific implementation of the load balancing algorithm involves calculating a comprehensive load score for each candidate server node. First, each indicator is normalized, converting it into a dimensionless value between 0 and 1. For example, CPU utilization percentage is directly divided by 100. Next, a corresponding configurable weight coefficient is assigned to each indicator (e.g., CPU weight 0.5, memory weight 0.3, network I / O weight 0.2) to reflect the degree of impact of different resources on service pressure. The comprehensive score of each candidate server node is calculated using a weighted summation formula: `Score = w_cpu * C + w_mem * M + w_io * I`, where C, M, and I are the normalized CPU, memory, and network I / O indicator values, respectively.

[0101] The candidate server node with the lowest score, considered to be the lightest currently loaded and with the greatest service potential, is selected as the application server to respond to upcoming preloaded query requests or high-concurrency data query requests. This decision-making process also considers node availability, such as filtering out nodes marked as offline or with failed health checks. The algorithm execution cycle can be synchronous (real-time calculation before each routing decision) or asynchronous (periodic calculation and caching of the optimal node list).

[0102] Next, the node selection results are implemented onto the actual network communication path, completing the targeted dispatch of the request. This targeted dispatch operation is achieved through a dynamic and updatable request routing layer. In the specific implementation, there are two main modes. In the first mode, a front-end soft load balancer (such as Nginx or LVS) is configured, and the decision result is transformed into a dynamic adjustment of the weights of the upstream server list of the load balancer or a direct specification of the target node. When a client (such as GS / CCS) initiates the preload query request or query request, the preload query request or query request first reaches the load balancer, which forwards it to the selected application server node according to the latest routing strategy (such as weighted least connections).

[0103] In the second decentralized model, routing logic is embedded in the client. The client SDK periodically (or via event subscription) obtains the currently optimal application server node address (IP and port) from the central configuration service. When a preload query request or query request needs to be initiated, the client directly establishes an RPC connection using the application server node address, thereby directing the preload query request or query request to the target node.

[0104] In summary, the implementation of the above-mentioned targeting patterns ensures that data access traffic is accurately directed to the lightest-loaded node in the cluster, making full use of the idle computing and memory resources of low-load servers to support caching services, and achieving intelligent allocation of system-level resources and proactive balancing of global load.

[0105] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the distributed data preloading and querying method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0106] This application provides a distributed data preloading and querying device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the distributed data preloading and querying method in the above embodiment 1.

[0107] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the distributed data preloading and querying device in the embodiments of this application. The distributed data preloading and querying device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The distributed data preloading and querying device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0108] like Figure 6 As shown, the distributed data preloading and querying device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the distributed data preloading and querying device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the distributed data preloading and querying device to communicate wirelessly or wiredly with other devices to exchange data. Although a distributed data preloading and querying device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0109] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0110] The distributed data preloading and querying device provided in this application, employing the distributed data preloading and querying method described in the above embodiments, can solve the technical problems of resource waste, complex architecture, and cache breakdown risk in existing caching schemes. Compared with the prior art, the beneficial effects of the distributed data preloading and querying device provided in this application are the same as those of the distributed data preloading and querying method provided in the above embodiments, and other technical features in this distributed data preloading and querying device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0111] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0113] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the distributed data preloading and querying method in the above embodiments.

[0114] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0115] The aforementioned computer-readable storage medium may be included in a distributed data preloading and querying device; or it may exist independently and not be assembled into a distributed data preloading and querying device.

[0116] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the distributed data preloading and querying device, the distributed data preloading and querying device implements the technical content of the distributed data preloading and querying method embodiment shown above.

[0117] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0120] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described distributed data preloading and querying method. This solves the technical problems of resource waste, complex architecture, and cache breakdown risk in existing caching schemes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the distributed data preloading and querying method provided in the above embodiments, and will not be repeated here.

Claims

1. A distributed data preloading and querying method, characterized in that, The distributed data preloading and querying method includes the following steps: Obtain historical query records and analyze the frequency and time distribution patterns of each query key based on the historical query records; Based on the frequency and time distribution pattern, one or more high-frequency query keys are identified from the historical query records; During a preset low-load period or after the application server starts, a preload query request based on the high-frequency query key is proactively sent to the database. Receive the preloaded query results returned by the database, and store the preloaded query results in advance in the cache area integrated by the application server; In response to a subsequent data query request, the data query request includes a query key; Based on the data query request, the cache area is searched to see if there is a query result that matches the query key; If the query result corresponding to the query key is found in the cache area, the query result is returned directly.

2. The distributed data preloading and querying method as described in claim 1, characterized in that, After the step of checking whether a query result matching the query key exists in the cache area based on the data query request, the method further includes: If the query result corresponding to the query key is not found in the cache area, determine whether there is a target query key that is the same as the query key in the database query being executed; If the target query key does not exist, a real-time query request based on the query key is initiated to the database, and the real-time query result returned by the database is stored in the cache area integrated by the application server.

3. The distributed data preloading and querying method as described in claim 2, characterized in that, After the step of determining whether a target query key identical to the query key exists in the database query being executed, the method further includes: If the target query key exists, then the current data query request is associated with the database query being executed corresponding to the target query key and registered accordingly. After the database query corresponding to the target query key is completed, the unified query result returned by the database will be distributed to all associated registered data query requests.

4. The distributed data preloading and querying method as described in claim 2, characterized in that, After the step of determining whether a target query key identical to the query key exists in the database query being executed, the method further includes: If the target query key does not exist, create a new entry based on the current query key in the pre-maintained incomplete query registration table; A corresponding database query is created based on the new entry tag.

5. The distributed data preloading and querying method as described in claim 1, characterized in that, Prior to the steps that respond to subsequent data query requests, the following are also included: Parse the target business data to be queried and extract the server identifier, business logic type identifier, and data entity identifier from the server cluster; Using predetermined encoding rules, the server identifier, business type identifier, and entity identifier are combined into a unique three-dimensional composite primary key, which serves as the query key.

6. The distributed data preloading and querying method as described in claim 1 or 2, characterized in that, The step of storing the query results in the cache area includes: The query results are serialized and converted into compact structured data blocks; The structured data blocks are associated with the corresponding query keys and stored using a binary large object format, and their storage timestamps are recorded.

7. The distributed data preloading and querying method as described in claim 6, characterized in that, After associating the structured data block with its corresponding query key and recording its storage timestamp, the method further includes: The cleanup process is triggered periodically or when the cache space reaches a threshold. Traverse the cache area, determine and clear expired query results based on the storage timestamp and predefined expiration policy; The expiration strategy configures different lifespan times for preloaded query results and real-time query results.

8. The distributed data preloading and querying method as described in any one of claims 1 to 7, characterized in that, The distributed data preloading and querying method also includes: Detect real-time load metrics of multiple candidate server nodes, including at least one of CPU utilization, memory utilization, and network I / O; According to the load balancing algorithm, the node with the lowest load is selected from the candidate server nodes as the application server to respond to the preloaded query request and the query request; The preloaded query request and the query request route are directed to the application server.

9. A distributed data preloading and querying device, characterized in that, The distributed data preloading and querying device stores a computer program, which, when executed by a processor, implements the distributed data preloading and querying method according to any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the distributed data preloading and querying method according to any one of claims 1-8.

Images