A data loading optimization method based on distributed cache

Abstract

The application discloses a data loading optimization method based on distributed cache and belongs to the field of data management. The specific steps comprise the following steps: a hierarchical architecture comprising a local cache, an edge cache and a central cache is constructed; a data request priority is calculated in real time based on a dynamic weight model, the model comprises five metrics, i.e. continuity, real-time degree, freshness, synchronization necessity and data hit degree, priority quantification is realized by combining metric weight and interaction item weight; when a client requests data, the local cache, the edge cache and the central cache are sequentially queried, an optimization strategy is triggered when the edge cache is not hit, the optimization strategy comprises automatic writing based on an admission threshold, active backfilling predicted by a deep learning model and slice traversal cleaning based on elimination priority when capacity is over limit. The application effectively improves the cache hit rate, reduces loading delay by dynamically adjusting the weight and intelligently optimizing the edge cache, and is suitable for high-frequency data access scenarios such as electronic whiteboards.

Description

Technical Field

[0001] This invention belongs to the field of data management, specifically relating to a data loading optimization method based on distributed caching. Background Technology

[0002] With the surge in data volume, efficient data storage and loading has become a key issue in the field of information technology. Especially in fast-response applications, data loading speed directly impacts performance and user experience. Distributed caching technology significantly improves data access speed by distributing data across multiple caching layers. In scenarios with frequent data access, such as electronic whiteboards, a well-designed distributed caching architecture is crucial.

[0003] Currently, distributed caching typically includes local caching, edge caching, and central caching. However, most solutions only employ simple caching strategies and lack intelligent optimization, resulting in low caching efficiency, high latency, and impacted performance.

[0004] Existing technologies have several problems: First, they fail to dynamically select the caching layer based on user request priority and data importance, causing unnecessary latency. Second, their cache update and loading strategies are too simplistic, failing to fully utilize the advantages of multi-layered caching, resulting in low hit rates. Third, insufficient data synchronization between edge and central caches cannot guarantee cache consistency under high concurrency. Therefore, existing technologies cannot meet the demands for efficient data loading and response speed. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the present invention aims to provide a data loading optimization method based on distributed caching. By designing a multi-level caching architecture, combining deep learning models to predict data access needs, dynamically adjusting the weights of different caching levels, and adopting an edge caching optimization strategy, the method effectively improves cache hit rate, reduces data loading latency, optimizes data request priority, and thereby enhances data loading efficiency.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A data loading optimization method based on distributed caching, comprising the following steps:

[0008] Step S1: Design a hierarchical caching architecture for whiteboard data, including local cache, edge cache and central cache;

[0009] Step S2: When a client has multiple data requests, the priority of the data requests is calculated in real time according to the preset dynamic weight model;

[0010] Step S3: When the client requests data, it first queries the local cache. If the data is found, it returns the data directly. If the data is not found, it queries the edge cache and the central cache in turn. If the data is not found in the edge cache, the edge cache optimization strategy is triggered.

[0011] Furthermore, the local cache, edge cache, and central cache each use an IndexedDB database, the edge cache uses a Redis cluster, and the central cache uses a Memcached cluster to handle different levels of data access requests.

[0012] Furthermore, the dynamic weighting model is used to calculate the data request priority of local cache, edge cache, and central cache;

[0013] The dynamic weighting model includes continuity. Real-time Freshness Synchronization Necessity and data hit rate Five metrics, including Z, with each metric having a maximum value of Z;

[0014] The formula for calculating the data request priority is as follows:

[0015]

[0016] in, This indicates the weight of the metric x; Interaction items that are based on continuity and real-time performance are considered as interaction item 1; The interaction item that relates to freshness and the necessity of synchronization is considered as interaction item 2; These are the weights of interaction item 1 and interaction item 2, respectively.

[0017] Furthermore, the calculation process for the continuity is as follows:

[0018] First, a step factor is defined based on the consecutive steps to which the i-th data block belongs. The formula for the step factor is:

[0019]

[0020] Where n represents the total number of consecutive steps;

[0021] Then, based on the velocity correction term and the acceleration correction term, the step factor is corrected to obtain the corrected value. The formula is as follows:

[0022]

[0023] in, It depends on the ratio of the user's drawing speed to the average drawing speed. When the user's drawing speed is greater than a preset multiple of the average drawing speed, Otherwise, it is 1; This is an acceleration correction term, determined by the ratio of user-drawn acceleration to average drawing acceleration. When the user-drawn acceleration is greater than the average drawing acceleration, when... Otherwise, it is 1; this is a speed correction term. It is a function for maximizing the value;

[0024] Finally, the continuity is obtained. The formula is as follows: It is a minimum value function;

[0025] Furthermore, the calculation process for the real-time degree is as follows:

[0026] First, preset corresponding base scores for different whiteboard operations. ;

[0027] Then, based on the overlap between the whiteboard interface focus and the current whiteboard view. The base score is adjusted to obtain the adjusted value. The formula is: ;

[0028] Ultimately, the real-time accuracy is obtained. The formula is .

[0029] Furthermore, the calculation process for freshness is as follows:

[0030] First, define the time decay factor. The formula is: ;in, , The time decay constant;

[0031] Then, define the operational correlation factor. The formula is: ;in, Let be the correlation between the i-th data block and the current operation;

[0032] Ultimately, freshness is achieved. The formula is: .

[0033] Furthermore, the calculation process for the synchronization necessity is as follows:

[0034] First, define the synchronization quantity factor. The formula is: Where k is the number of clients working synchronously;

[0035] Then, the network state correction factor is defined by the formula: Where b and d represent the current network bandwidth and latency, respectively. These represent historical average network bandwidth and latency, respectively.

[0036] Secondly, define the risk adjustment factor, with the following formula: ;in, This indicates the probability that the synchronized clients will modify the data simultaneously.

[0037] Finally, the synchronization necessity degree is obtained. The formula is: .

[0038] Furthermore, the calculation process for the data hit rate is as follows:

[0039] First, define the hit rate correction factor, the formula is: Where h represents the cache hit rate of the most recent fixed number of cache hits;

[0040] Then, the time fluctuation factor is defined by the formula: ;in, The standard deviation of response time This represents the average response time.

[0041] Finally, the data hit rate is obtained, and the formula is: .

[0042] Furthermore, the edge caching optimization strategy is implemented based on a deep learning model;

[0043] The edge caching optimization strategy process is as follows:

[0044] S301: Set an admission threshold. If the priority of the requested data is greater than the admission threshold, the edge cache will automatically write the data to the edge cache after obtaining the data from the central cache.

[0045] S302: Based on a deep learning model, predict the access probability, correlation, and synchronization necessity of data in the central cache. When the predicted access probability is greater than a preset threshold and meets any of the following conditions, write the corresponding data into the edge cache; the conditions include:

[0046] The correlation degree is greater than the preset correlation factor threshold;

[0047] The synchronization necessity is greater than the preset synchronization necessity threshold;

[0048] S303: When the edge cache capacity exceeds a preset capacity threshold, begin cleaning up data in the edge cache whose eviction priority exceeds the preset eviction threshold; the eviction priority is based on the continuity. Real-time Freshness Synchronization Necessity Calculations show that if the elimination priorities are the same, data with lower hit rate will be cleaned up.

[0049] Data with higher elimination priority is cleaned up first.

[0050] Furthermore, the deep learning model is a time series prediction algorithm based on LSTM networks, comprising: an input layer, an embedding layer, a two-layer LSTM module, a feature fusion layer, and an output layer.

[0051] Furthermore, the specific process of S303 is as follows:

[0052] S3031: When the edge cache capacity exceeds the preset capacity threshold, the cleanup process begins;

[0053] S3032: Eviction priority of pre-computed edge cache data;

[0054] S3033: Filter the traversal range; include data that has not been accessed within the preset recent time period into the traversal range; data whose real-time performance and synchronization necessity exceed the preset threshold are not included in the traversal range;

[0055] S3034: Perform shard traversal; edge cache is hash-sharded by data ID; each shard is traversed in descending order of eviction priority through a Redis sorted set, and data with eviction priority exceeding the threshold is cleaned up, with the amount of cleaned up data in each shard remaining equal; if the eviction priorities are the same, data with lower hit rate is cleaned up first.

[0056] S3035: Recalculate the edge cache capacity. If it is still greater than the preset capacity threshold, repeat steps S3033 and S3034.

[0057] S3036: When clearing cached data, check whether the edge cache is synchronized to the central cache; if not synchronized, upload the data to the central cache; if synchronized, delete it directly.

[0058] Therefore, the technical solution adopted by the present invention is as follows:

[0059] Compared with the prior art, the advantages of the present invention are as follows:

[0060] (1) The present invention designs a three-level caching architecture of local, edge and central, using IndexedDB, Redis cluster and Memcached cluster respectively to adapt to different data access requirements, reduce cross-layer access latency and improve basic data loading efficiency.

[0061] (2) The dynamic weight model proposed in this invention integrates five-dimensional metrics and interaction items, calculates request priority in real time, dynamically adjusts weights in combination with cache performance, accurately matches data importance, and improves cache hit rate and request processing targeting.

[0062] (3) The edge caching optimization strategy proposed in this invention is based on LSTM to predict data access characteristics, realize active backfilling and intelligent cleaning, prioritize the retention of high-value data, ensure the effectiveness of caching under limited capacity, and reduce the resource occupation of invalid data. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of the method of the present invention.

[0065] Figure 2 This is a diagram of the distributed caching architecture of the method of the present invention.

[0066] Figure 3 This is a flowchart of the edge caching optimization strategy of the method of the present invention.

[0067] Figure 4 This is a structural diagram of the LSTM prediction model of the method of the present invention. Detailed Implementation

[0068] To facilitate understanding by those skilled in the art, the present invention will be further described below in conjunction with embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0069] like Figure 1 As shown, the data loading optimization method based on distributed caching of the present invention includes the following steps:

[0070] Step S1: Design a hierarchical caching architecture for whiteboard data, including local cache, edge cache and central cache;

[0071] like Figure 2 As shown, a multi-layered distributed caching architecture was designed, including local cache, edge cache and central cache, to improve data loading efficiency and reduce access latency.

[0072] The local cache uses IndexedDB, a database that supports browser-side data storage, providing fast data access and efficient query operations, making it particularly suitable for storing user local data and temporary caches. The edge cache uses a Redis cluster, a high-performance in-memory data store with high throughput, low latency, and distributed capabilities, effectively supporting large-scale data storage and fast access, primarily caching frequently accessed data and latency-sensitive content. The central cache uses a Memcached cluster, a high-efficiency distributed in-memory cache suitable for storing and accessing large amounts of data, especially infrequently accessed data or globally shared static data. The workflow of this architecture is as follows: the client first queries the local cache; if a match is found, the data is returned directly; if not, the edge cache is queried; if the edge cache also fails to find the data, the central cache is requested.

[0073] Step S2: When a client has multiple data requests, the priority of the data requests is calculated in real time according to the preset dynamic weight model;

[0074] The dynamic weighting model is used to calculate the data request priority of local cache, edge cache, and central cache.

[0075] The dynamic weighting model includes continuity. Real-time Freshness Synchronization Necessity and data hit rate Five metrics, including Z, with each metric having a maximum value of Z;

[0076] In this embodiment, the upper limit of the value Z is set to 10;

[0077] The formula for calculating the data request priority is as follows:

[0078]

[0079] in, This indicates the weight of the metric x; Interaction items that are based on continuity and real-time performance are considered as interaction item 1; The interaction item that relates to freshness and the necessity of synchronization is considered as interaction item 2; These are the weights of interaction item 1 and interaction item 2, respectively.

[0080] The value ranges from 0.01 to 0.1, depending on the context.

[0081] The weight of the metric is adjusted in real time based on the total data hit rate, and the adjustment formula is as follows:

[0082]

[0083] For the next moment The increase or decrease;

[0084] After adjustment, all weights are normalized to generate new metric weights.

[0085] The calculation process for the continuity is as follows:

[0086] First, a step factor is defined based on the consecutive steps to which the i-th data block belongs. The formula for the step factor is:

[0087]

[0088] Where n represents the total number of consecutive steps;

[0089] Then, based on the velocity correction term and the acceleration correction term, the step factor is corrected to obtain the corrected value. The formula is as follows:

[0090]

[0091] in, It depends on the ratio of the user's drawing speed to the average drawing speed. When the user's drawing speed is greater than a preset multiple of the average drawing speed, Otherwise, it is 1; This is an acceleration correction term, determined by the ratio of user-drawn acceleration to average drawing acceleration. When the user-drawn acceleration is greater than the average drawing acceleration, when... Otherwise, it is 1; this is a speed correction term. It is a function for maximizing the value;

[0092] Finally, the continuity is obtained. The formula is as follows: It is a minimum value function.

[0093] In this embodiment, the user drawing speed v=Δd / Δt User-drawn acceleration a = Δv / Δt, When v is greater than 1.5 times the average drawing speed, =1.2; when a When the acceleration is greater than the average drawing acceleration ， =1.3；

[0094] In some embodiments, the user drawing speed and user drawing acceleration can be mapped based on a mapping function to obtain... and ;

[0095] The calculation process for the real-time performance is as follows:

[0096] First, preset corresponding base scores for different whiteboard operations. ;

[0097] In this embodiment, the whiteboard operation and corresponding basic score include:

[0098] Drawing points / lines / graphs: =10. Drag / Zoom: =9. Erase: =8. Text Input: =7. Auto-save / History: =3;

[0099] Then, based on the overlap between the whiteboard interface focus and the current whiteboard view. The base score is adjusted to obtain the adjusted value. The formula is: ;

[0100] Ultimately, the real-time accuracy is obtained. The formula is .

[0101] The calculation process for freshness is as follows:

[0102] First, define the time decay factor. The formula is: ;in, , t is the time decay constant; t is the current time. The time when the data was generated;

[0103] Then, define the operational correlation factor. The formula is: ;in, Let be the correlation between the i-th data block and the current operation;

[0104] The correlation is set to a preset value. For example, operations on the same layer, the same path, and the same graphic ID are set to correlation values ​​of 0.1, 0.5, and 0.9, respectively.

[0105] Ultimately, freshness is achieved. The formula is: .

[0106] The calculation process for the synchronization necessity is as follows:

[0107] First, define the synchronization quantity factor. The formula is: Where k is the number of clients working synchronously;

[0108] When k=1, it indicates a one-ended operation. =3; when k=2, it indicates a one-sided operation. =8; and so on.

[0109] Then, the network state correction factor is defined by the formula: Where b and d represent the current network bandwidth and latency, respectively. These represent historical average network bandwidth and latency, respectively.

[0110] Secondly, define the risk adjustment factor, with the following formula: ;in, This indicates the probability that the synchronized clients will modify the data simultaneously.

[0111] Finally, the synchronization necessity degree is obtained. The formula is: .

[0112] The calculation process for the data hit rate is as follows:

[0113] First, define the hit rate correction factor, the formula is: Where h represents the cache hit rate of the most recent fixed number of cache hits;

[0114] Then, the time fluctuation factor is defined by the formula: ;in, The standard deviation of response time This represents the average response time.

[0115] Finally, the data hit rate is obtained, and the formula is: .

[0116] For step 2, the following implementation method is given; for ease of understanding, the data is simplified.

[0117] The user draws line segments continuously on the lightweight whiteboard (data block 1 is the second point), simultaneously triggering the loading of historical graphics (data block 2 is the rectangle from 5 minutes ago). Assumptions: Local cache miss, edge cache request priority needs to be calculated; Dynamic weight model parameters: Z=10; Weights for each metric = [0.3, 0.3, 0.15, 0.1, 0.15], Interaction item weights α=0.02, β=0.01.

[0118] Data block 1:

[0119] Calculation step factor: consecutive steps n=2: ;

[0120] Calculation speed adjustment: For faster user rendering, the speed is 1.5 times faster than average.

[0121] Calculate the acceleration correction term: No acceleration. :

[0122] Calculate the correction value:

[0123] continuity

[0124] Operation type: Draw point, base score Ttype=10;

[0125] The user is currently focusing on the drawing area; interface focus overlap. =0.9:

[0126] Real-time ;

[0127] Generation time Δt = 10 seconds, decay constant τ = 30 seconds: ;

[0128] correlation , which are the continuous points of the current line segment;

[0129] Operational correlation factor ;

[0130] Freshness: ;

[0131] This data block is for one-ended operation k=1. ;

[0132] Network status: Historical average Historical average ;

[0133] Modification probability Single-ended conflict-free:

[0134] Synchronization Necessity ;

[0135] Edge cache hit rate h=0.8, response time , ;

[0136] Data hit rate ;

[0137] Priority score: .

[0138] Data block 2:

[0139] Calculation step factor: discontinuous steps n=1: ;

[0140] Speed ​​correction term: None.

[0141] Calculate the acceleration correction term: No acceleration. :

[0142] Calculate the correction value:

[0143] continuity

[0144] Operation type: View history, base score T type =3;

[0145] User not focusing on the historical area, interface focus overlap =0.2;

[0146] Real-time ;

[0147] Generation time Δt = 300 seconds, decay constant τ = 30 seconds;

[0148] correlation Weak association;

[0149] Freshness: ;

[0150] This data block is for multi-end operation k=3. ;

[0151] Network status: Historical average Historical average ;

[0152] Modification probability ;

[0153] Synchronization Necessity ;

[0154] Edge cache hit rate h=0.8, response time , Same as data block 1.

[0155] Data hit rate ;

[0156] Priority score: .

[0157] Step S3: When the client requests data, it first queries the local cache. If the data is found, it returns the data directly. If not, it queries the edge cache and the central cache sequentially. If the data is not found in the edge cache, the edge cache optimization strategy is triggered. Figure 3 As shown:

[0158] The edge caching optimization strategy is implemented based on a deep learning model;

[0159] The edge caching optimization strategy process is as follows:

[0160] S301: Set an admission threshold. If the priority of the requested data is greater than the admission threshold, the edge cache will automatically write the data to the edge cache after obtaining the data from the central cache.

[0161] Admission thresholds are typically determined based on factors such as the storage capacity of the edge cache and the data transfer cost between the central cache and the edge cache. For example, if the edge cache has a small storage capacity, to avoid the high transfer costs caused by frequently retrieving data from the central cache, the admission threshold will be set higher, allowing only very high-priority data to be written to the edge cache.

[0162] In this embodiment, the admission threshold is set to 10;

[0163] S302: Based on a deep learning model, predict the access probability, correlation, and synchronization necessity of data in the central cache. When the predicted access probability is greater than a preset threshold and meets any of the following conditions, write the corresponding data into the edge cache; the conditions include:

[0164] The correlation degree is greater than the preset correlation factor threshold;

[0165] The synchronization necessity is greater than the preset synchronization necessity threshold;

[0166] The preset threshold for access probability, the threshold for correlation factor, and the threshold for synchronization necessity are set based on the same considerations as the admission threshold, which are determined by factors such as the storage capacity of the edge cache and the data transmission cost between the central cache and the edge cache.

[0167] In this embodiment, the preset threshold is set to 0.7; the correlation factor threshold and the synchronization necessity threshold are both set to 6.

[0168] A higher threshold value can reduce the frequency of data exchange between the central cache and the edge cache.

[0169] S303: When the edge cache capacity exceeds a preset capacity threshold, begin cleaning up data in the edge cache whose eviction priority exceeds the preset eviction threshold; the eviction priority is based on the continuity. Real-time Freshness Synchronization Necessity Calculations show that if the elimination priorities are the same, data with lower hit rate will be cleaned up.

[0170] Data with higher elimination priority is cleaned up first.

[0171] The preset capacity threshold is generally set to 80%, that is, 20% of the edge cache is reserved to ensure fast data exchange.

[0172] The preset elimination threshold, based on the numerical distribution of elimination priorities in the current edge cache, ensures that the 10% of data with the highest elimination priority can be cleaned up by passing the elimination threshold.

[0173] In this embodiment, the formula for calculating the elimination priority is as follows:

[0174]

[0175] The specific process of S303 is as follows:

[0176] S3031: When the edge cache capacity exceeds the preset capacity threshold, the cleanup process begins;

[0177] S3032: Eviction priority of pre-computed edge cache data;

[0178] S3033: Filter the traversal range; include data that has not been accessed within a preset time period into the traversal range; data whose real-time performance and synchronization necessity exceed a preset threshold are not included in the traversal range; this step can shorten the calculation time;

[0179] S3034: Perform shard traversal; edge cache is hash-sharded by data ID; each shard is traversed in descending order of eviction priority through a Redis sorted set, and data with eviction priority exceeding the threshold is cleaned up, with the amount of cleaned up data in each shard remaining equal; if the eviction priorities are the same, data with lower hit rate is cleaned up first.

[0180] S3035: Recalculate the edge cache capacity. If it is still greater than the preset capacity threshold, repeat steps S3033 and S3034.

[0181] S3036: When clearing cached data, check whether the edge cache is synchronized to the central cache; if not synchronized, upload the data to the central cache; if synchronized, delete it directly.

[0182] A time-series forecasting algorithm using an LSTM network is employed to analyze historical data request records and predict future access demands based on this data. This prediction intelligently determines which data should be prioritized for backfilling into the edge cache, thereby improving cache hit rate and data loading efficiency.

[0183] like Figure 4 As shown, the deep learning model uses an LSTM network time series prediction algorithm, which includes: an input layer, an embedding layer, a two-layer LSTM module, a feature fusion layer, and an output layer.

[0184] The input layer receives historical data request records from clients, including access time, request type, whiteboard operation, and current cache hit rate. This input data is transformed into time series data, providing the foundation for training and prediction of the LSTM model. When feeding time series data to the LSTM model, the data is formatted as a multidimensional array, where each dimension represents a feature, and each time step represents a specific feature of a request.

[0185] In its implementation, the embedding layer maps each user type, request type, time period, and current cache hit rate into a low-dimensional vector space. Each feature corresponds to an embedding matrix, where each row of the matrix represents a vector corresponding to a feature category. The output of the embedding layer is a sequence of vectors consisting of multiple embedding vectors, which serve as input to the LSTM network, helping the model capture the relationship between different features and future access needs.

[0186] The first layer of the two-layer LSTM module is mainly responsible for handling short-term dependencies in the data. In this layer, each time step of the LSTM depends on the feature vector transformed by the embedding layer and the hidden state of the previous time step. and unit status This determines whether new information needs to be added to the current cell state. The formula is as follows:

[0187]

[0188] in, These are the input gate, the forget gate, and the output gate. It is a unit state. It is in a hidden state. This is the current input, and W and b are the weights and biases, respectively. It is the sigmoid activation function, and tanh is the hyperbolic tangent function.

[0189] The second LSTM layer primarily handles long-term dependencies in the data. This is achieved by processing the output of the first LSTM layer. The data is then passed to the second LSTM layer, enabling the learning of long-term temporal features across time steps. This is crucial for predicting periodic changes and long-term trends in access demand. The computation of the second LSTM layer is similar to that of the first layer, as shown in the following formula:

[0190]

[0191] In this layer, all symbols have the same meaning as in the first LSTM layer, but the weights and biases are parameters of the second LSTM layer. The main task of this LSTM layer is to process the hidden state information from the first layer and capture long-term dependencies across time steps to reflect long-term patterns such as user behavior cycles.

[0192] Based on the output of the two-layer LSTM module, feature fusion is performed. Feature fusion integrates temporal features and cache state information extracted from the LSTM module to enhance the predictive ability of future data access needs. By fusing multiple temporal features from the LSTM module, the priority of data backfilling can be evaluated more comprehensively, further optimizing the data loading strategy of the edge cache. For example, suppose the output of the second-layer LSTM is... It combines this with factors such as current access frequency to generate a comprehensive feature representation:

[0193]

[0194] in, It comes from other characteristics, such as the current access frequency, etc. It is a fusion operation, which means weighted summation.

[0195] The core task of the final output layer of the LSTM network is to output the final prediction result based on the previously processed features from the two-layer LSTM and fusion. The prediction result includes the data access probability, correlation, and synchronization necessity.

[0196] The input to the output layer comes from the feature fusion module. The output layer is the second LSTM layer, which handles long-term dependencies in the data, while the feature fusion module integrates additional feature information. Together, these outputs provide the necessary contextual information for predicting future data access needs. The output layer typically begins by linearly transforming the input features; weight matrices and bias terms are used to map the input features to a space more suitable for prediction. This transformation can be expressed as:

[0197] ;

[0198] in, It is the feature representation output from the output layer. It is a weight matrix. It is a bias term. It is the fused feature vector.

[0199] After the linear transformation, an activation function, such as the sigmoid function, is applied to the output layer to ensure that the output value conforms to the expected range and to improve the model's non-linear fitting ability. The activation function formula is:

[0200]

[0201] in, It is the sigmoid activation function. This is the prediction result.

[0202] Output of the output layer This represents a prediction of future data access needs. In this embodiment, the prediction results include three types: the access probability of the data request, the relevance, and the synchronization necessity.

[0203] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data loading optimization method based on distributed caching, characterized in that, The method includes the following steps: Step S1: Design a hierarchical caching architecture for whiteboard data, including local cache, edge cache and central cache; Step S2: When a client has multiple data requests, the priority of the data requests is calculated in real time according to the preset dynamic weight model; Step S3: When the client requests data, it first queries the data from the local cache; If the data matches, return the data directly. If the data is not found, the edge cache and the central cache are queried in turn. If the data is not found in the edge cache, the edge cache optimization strategy is triggered. The local cache, edge cache, and central cache all use IndexedDB databases, the edge cache uses a Redis cluster, and the central cache uses a Memcached cluster to handle different levels of data access requests. The dynamic weighting model is used to calculate the data request priority of local cache, edge cache, and central cache. The dynamic weighting model includes continuity. Real-time Freshness Synchronization Necessity and data hit rate Five metrics, including [list of metrics], each with a maximum numerical value. ; The formula for calculating the data request priority is as follows: in, , Representing measurement The weights; Interaction items that are based on continuity and real-time performance are considered as interaction item 1; The interaction item that relates to freshness and the necessity of synchronization is considered as interaction item 2; and These are the weights of interaction item 1 and interaction item 2, respectively. The weights of all the metrics are adjusted in real time based on the overall hit rate of the data; The edge caching optimization strategy is implemented based on a deep learning model; The edge caching optimization strategy process is as follows: S301: Set an admission threshold. If the priority of the requested data is greater than the admission threshold, the edge cache will automatically write the data to the edge cache after obtaining the data from the central cache. S302: Based on a deep learning model, predict the access probability, correlation, and synchronization necessity of data in the central cache. When the predicted access probability is greater than a preset threshold and meets any of the following conditions, write the corresponding data into the edge cache; the conditions include: The correlation degree is greater than the preset correlation factor threshold; The synchronization necessity is greater than the preset synchronization necessity threshold; S303: When the edge cache capacity exceeds a preset capacity threshold, begin cleaning up data in the edge cache whose eviction priority exceeds the preset eviction threshold; the eviction priority is based on the continuity. Real-time Freshness Synchronization Necessity Calculations show that if the elimination priorities are the same, data with low hit rate will be cleaned up. Data with higher elimination priority is cleaned up first; The specific process of S303 is as follows: S3031: When the edge cache capacity exceeds the preset capacity threshold, the cleanup process begins; S3032: Eviction priority of pre-computed edge cache data; S3033: Filter the traversal range; include data that has not been accessed within a preset time period into the traversal range; data whose real-time performance and synchronization necessity exceed a preset threshold are not included in the traversal range; S3034: Perform shard traversal; edge cache is hash-sharded by data ID; each shard is traversed in descending order of eviction priority through a Redis sorted set, and data with eviction priority exceeding the threshold is cleaned up, with the amount of cleaned up data in each shard remaining equal; if the eviction priorities are the same, data with low hit rate is cleaned up first. S3035: Recalculate the edge cache capacity. If it is still greater than the preset capacity threshold, repeat steps S3033 and S3034. S3036: When clearing cached data, check whether the edge cache is synchronized to the central cache; if not synchronized, upload the data to the central cache; if synchronized, delete it directly.

2. The data loading optimization method based on distributed caching according to claim 1, characterized in that, The calculation process for the continuity is as follows: First, a step factor is defined based on the consecutive steps to which the i-th data block belongs. The formula for the step factor is: Where n represents the total number of consecutive steps; Then, based on the velocity correction term and the acceleration correction term, the step factor is corrected to obtain the corrected value. The formula is as follows: in, This is a speed adjustment factor, depending on the user's drawing speed; This is an acceleration correction term, depending on the acceleration drawn by the user; It is a function for maximizing the value; Finally, the continuity is obtained. The formula is as follows: ; It is a minimum value function; The calculation process for the real-time performance is as follows: First, preset corresponding base scores for different whiteboard operations. ; Then, based on the overlap between the whiteboard interface focus and the current whiteboard view. The base score is adjusted to obtain the adjusted value. The formula is: ; Ultimately, the real-time accuracy is obtained. The formula is .

3. The data loading optimization method based on distributed caching according to claim 1, characterized in that, The calculation process for freshness is as follows: First, define the time decay factor. The formula is: ;in, , The time decay constant; For the current time, The time when the data was generated; Then, define the operational correlation factor. The formula is: ;in, Let be the correlation between the i-th data block and the current operation; Ultimately, freshness is achieved. The formula is: ; The calculation process for the synchronization necessity is as follows: First, define the synchronization quantity factor. The formula is: Where k is the number of clients working synchronously; Then, the network state correction factor is defined by the formula: Where b and d represent the current network bandwidth and latency, respectively. and These represent historical average network bandwidth and latency, respectively. Secondly, define the risk adjustment factor, with the following formula: ;in, This indicates the probability that the synchronized clients will modify the data simultaneously. Finally, the synchronization necessity degree is obtained. The formula is: .

4. The data loading optimization method based on distributed caching according to claim 1, characterized in that, The calculation process for the data hit rate is as follows: First, define the hit rate correction factor, the formula is: ;in, This indicates the cache hit rate over a fixed number of recent preset cycles; Then, the time fluctuation factor is defined by the formula: ;in, The standard deviation of response time This represents the average response time. Finally, the data hit rate is obtained, and the formula is: .

5. The data loading optimization method based on distributed caching according to claim 1, characterized in that, The deep learning model described is a time series prediction algorithm based on LSTM networks, comprising: an input layer, an embedding layer, a two-layer LSTM module, a feature fusion layer, and an output layer.

Images