Data cache management method and system based on big data

Abstract

The invention discloses a data cache management method and system based on big data, and relates to the technical field of data cache management, and the method comprises the steps: collecting access mode metadata, constructing a data cache state space and an action space, and screening an effective action set of data cache; on the basis of the state space, calculating a context feature weight, and constructing a CO-IAM model to predict a data caching action; performing data caching action implementation based on a CO-IAM model prediction result; by constructing a state space and an action space of data caching, introducing a CO-IAM model, performing precise modeling and dynamic adjustment based on context feature weight, generating an invalid action mask, filtering useless caching actions, optimizing a caching hit rate and access delay, and improving the performance of the whole caching system.

Description

Technical Field

[0001] The present invention relates to the technical field of data cache management, and in particular to a data cache management method and system based on big data. Background Art

[0002] With the rapid development of information technology and the widespread popularization of Internet applications, big data technology has emerged. Academia and industry have proposed LRU strategies and LFU strategies based on access frequency and cache replacement methods based on heuristic learning, applying artificial intelligence and machine learning technologies to data caching. Through deep learning models, cache hit rates are predicted and data caching strategies are optimized to further improve the intelligent management level of cache.

[0003] Existing data cache management methods still have shortcomings. Although some current machine learning-based caching methods have introduced intelligent prediction mechanisms, most of them have not fully integrated the correlation between contextual information and access pattern metadata, resulting in limited accuracy and real-time performance of cache action predictions. Existing technologies generally lack effective mechanisms for identifying and eliminating invalid cache actions, which easily introduces redundant operations and wastes computing resources. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a data cache management method and system based on big data to solve the problem that although some current machine learning-based caching methods have introduced intelligent prediction mechanisms, most of them have not fully integrated the correlation between context information and access pattern metadata, resulting in limited accuracy and real-time performance of cache action predictions. The existing technology generally lacks an effective identification and elimination mechanism for invalid cache actions, which easily introduces redundant operations and wastes computing resources.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a data cache management method based on big data, which comprises: Collect access pattern metadata, including user access logs and system status data, build data cache state space and action space, and filter the effective action set for data cache; Based on the state space, the context feature weights are calculated and the CO-IAM model is constructed to predict data caching actions; Implement data caching actions based on the CO-IAM model prediction results; Build a visual interface to display the growth rate of system status data after the action is implemented, and store the access pattern metadata generated by the analysis.

[0007] As a preferred solution of the data cache management method based on big data described in the present invention, wherein: The collecting of access pattern metadata and the construction of data cache state space and action space include: Configure an Apache Flink cluster, define the Flink data stream time step window W, and aggregate and collect user access logs and system status data. The user access log data includes data ID, time stamp, user ID and context information; The context information includes the type of data, the business scenario of the data, and the type of user; The system status data includes system load, cache hit rate and cache capacity of each layer; Group user access log data by user ID, record the data ID accessed by each user in the time step window W, extract the context signal data of each recorded data ID, convert the context signal data of each data ID accessed by each user and each recorded data ID in the time step window W into text, use it as the input of the pre-trained BERT model, and output the user behavior vector and data vector; Calculate the number of accesses to the data ID in the time step window W, calculate the cosine similarity between the user behavior vector and the data vector, and assign a priority score to the data of each scenario based on expert experience; Build a Transformer model, collect historical user access log data, convert it into text, and then train the Transformer model. Then, convert the user access log data into text and input it into the trained Transformer model to predict the data access probability. The number of accesses of the data ID in the time step window W, the cosine similarity between user behavior and data, the data allocation priority score of each scenario, and the predicted access probability are constructed as a feature vector; The system state data is standardized, and the standardized system state data and the characteristic vector are combined into a state space to obtain a state vector; The cache capacity change, cache data migration and switching replacement strategy are defined as three actions in the action space to construct the action space.

[0008] As a preferred solution of the data cache management method based on big data described in the present invention, the effective action set for filtering the data cache refers to calculating the access probability threshold based on the system load of the current time step, and setting the filtering rules for the effective actions to filter the effective action set.

[0009] As a preferred solution of the data cache management method based on big data of the present invention, the calculation of context feature weights includes: Use the pre-trained Word2Vec model to convert user access log data into continuous feature vectors to obtain context features; Normalize the state space data, calculate the business scenario priority score, use the ratio method to calculate the user type request ratio, and use the maximum normalization method to calculate the priority weight and type ratio weight based on the user type request ratio and business scenario priority score. Calculate the Pearson correlation coefficient between the proportion of user type requests, business scenario priority score, and cache hit rate changes. ,Based on the priority weight and type proportion weight, the interaction term is introduced to calculate the context feature weight.

[0010] As a preferred solution of the data cache management method based on big data of the present invention, the construction of the CO-IAM model to predict data cache actions includes: Collect historical state data to perform initial training on the CO-IAM model and input the state vector into the trained CO-IAM model; The self-attention layer is used to extract the dependency of the state vector using the self-attention mechanism and perform weighted summation to obtain an enhanced state; The multi-layer perceptron (MLP) is used to map the enhanced state to the action space and generate logits; The invalid action mask generation unit is used to generate a mask vector based on the valid action set, perform a dot product operation on the mask vector of each action and the generated logits, and output the result as the initial action probability through the softmax function; The incremental data processing unit is used to add access pattern metadata of the newly added time step; The composite objective optimization unit is configured to calculate a basic reward using a weighted sum method based on a cache hit rate and a system load change in the normalized state space data, and adjust the basic reward in combination with a context weight and an action priority; Define the value network, use temporal difference (TD) estimation to calculate the advantage value, use the Actor-Critic algorithm to calculate the actor loss and critic loss based on the advantage value and the enhanced state, and take the weighted sum of the actor loss and the critic loss as the composite loss; With the goal of minimizing the composite loss, the gradient descent method is used to update the policy parameters using the actor loss and the value network parameters using the critic loss. The context weight adjustment unit is used to adjust the initial action probability using the context weight and output the final data cache action probability; The distributed training and reasoning framework allocates nodes for model training and reasoning based on dynamic load scheduling and performs parallel computing.

[0011] As a preferred solution of the data cache management method based on big data of the present invention, the data cache action is implemented based on the prediction result of the CO-IAM model, including: Based on the predicted action probability of the CO-IAM model, if the action with the maximum predicted probability is the migration action, calculate the migration target and cost, and perform data migration; If the maximum action prediction probability is the adjusted capacity, calculate the new memory cache capacity and compare whether the new memory cache capacity is greater than the maximum memory cache capacity. If the new memory cache capacity is less than or equal to the maximum memory cache capacity, cache the newly added data. If the new memory cache capacity is larger than the maximum memory cache capacity, hot data will be allocated first until the memory cache is exhausted, and the remaining allocatable capacity will be incrementally allocated to the distributed cache; If the maximum action prediction probability is the switch replacement strategy, calculate the data replacement priority and first delete the data with the lowest priority score.

[0012] As a preferred solution of the data cache management method based on big data of the present invention, wherein: the construction of a visual interface to display the growth rate of system status data after the action is implemented and the storage of access pattern metadata generated by the analysis include: Calculate the relative growth rate of system status data after the action is executed, and use React.js to build a visual interface for display; The access pattern metadata generated by the analysis is stored in the database, and security access measures are set. The database backs up the access pattern metadata to the cloud and regularly performs integrity checks on the stored data and backup data. After the test is completed, an integrity test record is generated and stored synchronously in the database.

[0013] In a second aspect, the present invention provides a data cache management system based on big data, comprising: The collection and construction module is used to collect access pattern metadata, including user access logs and system status data, to build the data cache state space and action space, and to filter the effective action set of the data cache; The calculation and prediction module is used to calculate the context feature weights based on the state space and build the CO-IAM model to predict data caching actions; Implementation module, used to implement data caching actions based on the prediction results of the CO-IAM model; The visualization storage module is used to build a visualization interface to display the growth rate of system status data after the action is implemented, and to store the access pattern metadata generated by the analysis.

[0014] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the big data-based data cache management method according to the first aspect of the present application.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any step of the big data-based data cache management method according to the first aspect of the present application.

[0016] The present application has the following beneficial effects: the present application collects access pattern metadata, including user access logs and system state data, constructs a data cache state space and an action space, and screens an effective action set of the data cache; based on the state space, the present application calculates a context feature weight, constructs a CO-IAM model to predict a data cache action, and performs the data cache action based on the prediction result of the CO-IAM model; the present application constructs a visual interface to display a growth rate of system state data after the action is performed, stores the generated access pattern metadata, improves the intelligent level of the data cache strategy, improves the prediction accuracy of the data cache management, and enhances the scalability of the cache management system. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0018] Figure 1 The flowchart of the big data-based data cache management method in embodiment 1.

[0019] Figure 2 The schematic diagram of the big data-based data cache management system in embodiment 1.

[0020] Figure 3 The CO-IAM model structure diagram in embodiment 1.

[0021] Figure 4 The cache action implementation flowchart in embodiment 1. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0025] Example 1, with reference to Figures 1 to 4 , which is the first embodiment of the present invention, provides a data cache management method based on big data, including the following steps: S1. Collect access pattern metadata, including user access logs and system status data, build data cache state space and action space, and filter the effective action set for data cache; Preferably, configure an Apache Flink cluster, define a Flink data stream time step window W, and aggregate and collect user access logs and system status data; The user access log data includes data ID, time stamp, user ID and context information; The context information includes the data type, the data's business scenario (recommendation, search, and payment), and the user type (regular user and VIP user); The system status data includes system load (including CPU, memory, and bandwidth utilization), cache hit rate for each layer (memory, distributed, and disk), and cache capacity (the current allocation ratio of each layer of cache); Group user access log data by user ID, record the data ID accessed by each user in the time step window W, and extract the context signal data of each recorded data ID; Convert the context signal data of each user-accessed data ID and each recorded data ID in the time step window W into text as the input of the pre-trained BERT model, and output the user behavior vector and data vector; Calculate the number of accesses to the data ID in the time step window W, calculate the cosine similarity between the user behavior vector and the data vector, and assign a priority score to the data of each scenario based on expert experience; The Transformer model is constructed, historical user access log data is converted into text, and the Transformer model is trained, the user access log data is converted into text and input into the trained Transformer model, and the data access probability is predicted; The accessed times of data IDs in the time step window W, the cosine similarity between user behavior and data, the data allocation priority score of each scene, and the predicted access probability are constructed as a feature vector; The system state data is standardized, and the standardized system state data and the feature vector are merged into a state space to obtain a state vector. The cache capacity change size, cache data migration, and switching replacement strategy are defined as three actions of the action space.

[0026] By calculating the cosine similarity of data access times and user behavior vectors and data vectors, and assigning data priority scores based on expert experience, three types of indicators with different properties are combined, including statistical features (access times), semantic features (cosine similarity), and expert experience (priority score). The value of data access is comprehensively evaluated and measured, and by constructing access times, cosine similarity, priority score, and access probability into a feature vector, standardizing system state data, and merging it with a feature vector into a state space, a state vector is obtained. The state vector considering system load and data features can fully reflect the current system running state and cache environment.

[0027] Further, the access probability threshold is calculated based on the system load of the current time step, formula: , Where, is the access probability threshold at time step t, is the adjustment coefficient, set using the experimental method, is the system load at time step t, is the maximum system load. Based on the access probability threshold, set the effective action set, formula: , Where, is the effective action set, is a single action, which is the index of the action, is the complete action space, is the predicted probability of data being accessed, is the ith data, is the cache capacity of the kth layer, is the minimum cache capacity.

[0028] By considering the time-varying characteristics of system load, the access probability threshold is calculated, which avoids the inadaptability problem caused by the fixed threshold. By introducing the cache capacity constraint and the access prediction probability setting effective action set, the system stability is guaranteed while the cache efficiency and access performance are greatly improved, and the overhead of data migration and resource scheduling is reduced.

[0029] S2, based on the state space, calculate the context feature weight, and build a CO-IAM model to predict data cache action; Preferably, the pre-trained Word2Vec model is used to convert the user access log data (discrete data) into a continuous feature vector to obtain the context feature. The data in the state space is normalized, and the business scenario priority score is calculated, formula: , Wherein, is the priority score of the business scenario, is the normalized scenario request frequency, is the scenario request frequency, is the maximum scenario frequency, is the time step attenuation coefficient, and the experimental method is set to is the data freshness, which represents the difference between the current time step t and the scene data time step; The ratio method is used to calculate the user type request proportion, and based on the user type request proportion and the business scenario priority score, the maximum value normalization method is used to calculate the priority weight and the type proportion weight (comparing the user type request proportion and the business scenario priority score, and selecting the larger one for normalization processing); In time step t, the Pearson correlation coefficient of the change of user type request proportion and business scenario priority score and cache hit rate is calculated , based on the priority weight and the type proportion weight, the interaction term is introduced, and the context feature weight is calculated , formula: , Wherein, is the change of cache hit rate, is the interaction term coefficient, is the maximum value selected for the interaction term coefficient, which is set based on the actual situation, is the scaling factor, which is set using the experimental method, is the adjustment parameter, which is set using the experimental method, is the type of user.

[0030] By normalizing state space data and calculating a priority score based on scenario request frequency, time step decay, and data freshness, the importance and timeliness of business scenarios are dynamically reflected, which helps in resource scheduling and decision optimization. The linear relationship between user behavior, scenario characteristics, and cache hit rate is characterized by the Pearson correlation coefficient, and the context feature weights are calculated based on weighted interaction terms to quantify the actual impact of features on system performance. The theoretical indicators are organically combined with the actual system performance to make the feature weights closer to the actual effects.

[0031] Furthermore, a CO-IAM model was constructed, which includes a self-attention layer, a 5-layer multi-layer perceptron (MLP) (layer sizes are 512, 256, 128, 64, action space sizes, each of the first four layers uses a ReLU activation function, and the last layer has a linear output), an invalid action mask generation unit, an incremental data processing unit, a compound objective optimization unit, a context weight adjustment unit, and a distributed training and inference framework; Collect historical state data to perform initial training on the CO-IAM model and input the state vector into the trained CO-IAM model; The self-attention layer is used to extract the dependency of the state vector using the self-attention mechanism and perform weighted summation to obtain an enhanced state; The 5-layer multi-layer perceptron (MLP) is used to map the enhanced state to the action space and generate logits; The invalid action mask generating unit is used to generate a mask vector based on the valid action set, and the formula is: , in, For the The mask value of each action; Perform a dot product operation on the mask vector of each action and the generated logits, and output the result as the initial action probability through the softmax function; The incremental data processing unit is used to add access pattern metadata for the newly added time step, generate reinforcement learning triples (state, action, and reward), and perform preprocessing (normalization, discretization, and reward adjustment) to generate data required by the model; The composite objective optimization unit is used to calculate the basic reward using a weighted sum method based on the cache hit rate and system load change in the normalized state space data, and adjust the basic reward in combination with the context weight and action priority. The formula is: , in, To adjust the rewards, As a basic reward, For action Priority (set to 1 if it involves VIP users, otherwise 0); The value network is defined to share the first four layers with the MLP. The last layer outputs the state value and uses the temporal difference (TD) estimation to calculate the advantage value. The formula is: , in, is the advantage value, For the time step The value of the enhanced state, is the parameter of the value network; Based on the advantage value and the enhanced state, the actor loss and the critic loss are calculated separately using the Actor-Critic algorithm, and the weighted sum of the actor loss and the critic loss is used as the composite loss; With the goal of minimizing the composite loss, the gradient descent method is used to update the policy parameters using the actor loss and the value network parameters using the critic loss. The context weight adjustment unit is used to adjust the initial action probability using the context weight, formula: , in, For the adjusted state at time step t, given the enhanced state and strategy parameters In case of action The probability of being selected, At time step t, given the enhanced state and strategy parameters In case of action The probability of being selected, For action Priority (set to 1 if it involves VIP users, otherwise 0); The distributed training and reasoning framework allocates nodes for model training and reasoning based on dynamic load scheduling and performs parallel computing.

[0032] Through the self-attention mechanism, the internal dependencies of the state vector are captured, and important feature dimensions are automatically focused on. Through the deep neural network (MLP) layer-by-layer abstraction and mapping, complex state-action relationships are learned, a more reasonable action strategy distribution is generated, and the system decision-making ability is improved. By dynamically generating mask vectors, unavailable actions in the current environment are shielded to avoid invalid decisions. The incremental data processing unit processes new access pattern data, updates learning data in real time, and ensures that the model is synchronized with the business environment. Through weighted processing of cache hit rate, system load and action priority, compound rewards are calculated to ensure cache efficiency and load balance. Through the Actor-Critic mechanism, strategy and value estimation are optimized separately, and the advantage value is used to balance the two to achieve efficient strategy learning and value estimation, improve sample utilization efficiency, and make the strategy more sensitive to business scenarios by further integrating contextual information. Through the distributed mechanism, computing resources are allocated according to load conditions to improve parallelism and resource utilization.

[0033] S3. Implement data caching actions based on the prediction results of the CO-IAM model; Preferably, based on the predicted action probability of the CO-IAM model, if the maximum action prediction probability is the migration action, calculate the migration target and cost, formula: , , in, For data The cost, For data At time step The predicted probability of being visited, is the hotspot data threshold, which is set based on historical regression estimation. The minimum value allowed for the memory cache capacity. For data The migration cost, is the network bandwidth utilization rate, is the data size (normalized), The fixed migration delay is determined by the device used. is the maximum cost threshold; like , cut data from the distributed cache To the memory cache, if , cut data from the memory cache To distributed cache; If the maximum action prediction probability is adjusted, calculate the new memory cache capacity using the formula: , in, is the new memory cache capacity, is the original memory cache capacity, is the allocatable capacity increment, is the total predicted probability of hotspot data; Compare the new memory cache capacity to see if it is greater than the maximum memory cache capacity. If the new memory cache capacity is less than or equal to the maximum memory cache capacity, cache the newly added data. If the new memory cache capacity is larger than the maximum memory cache capacity, hot data will be allocated first until the memory cache is exhausted, and the remaining allocatable capacity will be incrementally allocated to the distributed cache; If the maximum action prediction probability is the switching replacement strategy, calculate the data replacement priority, formula: , in, For data The replacement priority score, is the predicted probability weight, and are the mean and maximum values ​​of the predicted data access probability, is the mean and maximum value of the access probability of historical data, For historical data The frequency of visits, is the context feature value; The data with the lowest priority score is deleted first.

[0034] By predicting action probabilities and performing migration actions, the adaptive adjustment of the cache system can be effectively achieved, manual intervention can be reduced, frequent and costly data migration can be avoided, and performance and resource consumption can be balanced. By automatically calculating the new memory cache capacity, the capacity can be adjusted based on dynamic hot data to achieve hierarchical cache optimization. By calculating the data replacement priority score, it can dynamically adapt to changes in data access patterns, maintain a high cache hit rate and stability, and improve the system's long-term self-optimization capabilities.

[0035] S4. Build a visual interface to display the growth rate of system status data after the action is implemented, and store the access pattern metadata generated by the analysis; Preferably, the relative growth rate of the system status data after the action is executed is calculated and a visual interface is constructed using React.js for display; The access pattern metadata generated by the analysis is stored in the database, and security access measures are set. The access pattern metadata is backed up in the cloud, and the integrity of the stored data and backup data is regularly checked. After the test is completed, the integrity test record is generated and stored synchronously in the database.

[0036] By quantitatively analyzing system change trends and displaying them through an intuitive interface, the efficiency and accuracy of data decision-making are improved. Through cloud-based backup of access mode metadata and regular integrity testing, high data availability and consistency are guaranteed. By automatically generating and storing integrity test records, data security capabilities are enhanced and compliance audit needs are supported.

[0037] This embodiment also provides a data cache management system based on big data, including: The collection and construction module is used to collect access pattern metadata, including user access logs and system status data, to build the data cache state space and action space, and to filter the effective action set of the data cache; The calculation and prediction module is used to calculate the context feature weights based on the state space and build the CO-IAM model to predict data caching actions; Implementation module, used to implement data caching actions based on the prediction results of the CO-IAM model; The visualization storage module is used to build a visualization interface to display the growth rate of system status data after the action is implemented, and to store the access pattern metadata generated by the analysis.

[0038] This embodiment also provides a computer device, which is suitable for the data cache management method based on big data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the data cache management method based on big data proposed in the above embodiment.

[0039] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0040] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the data cache management method based on big data proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0041] In summary, the present invention collects access pattern metadata, including user access logs and system status data, constructs a data cache state space and action space, and filters a valid action set for data cache; calculates context feature weights based on the state space, constructs a CO-IAM model to predict data cache actions; implements data cache actions based on the prediction results of the CO-IAM model; constructs a visual interface to display the growth rate of system status data after the action is implemented, and stores the access pattern metadata generated by the analysis, thereby improving the intelligence level of the data cache strategy, enhancing the prediction accuracy of data cache management, and enhancing the scalability of the cache management system.

[0042] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data cache management method based on big data, characterized by: include, Collect access pattern metadata, including user access logs and system status data, build data cache state space and action space, and filter the effective action set for data cache; Based on the state space, the context feature weights are calculated and the CO-IAM model is constructed to predict data caching actions; The CO-IAM model includes a self-attention layer, a multi-layer perceptron (MLP), an invalid action mask generation unit, an incremental data processing unit, a composite target optimization unit, a context weight adjustment unit, and a distributed training and inference framework. Implement data caching actions based on the CO-IAM model prediction results; Build a visual interface to display the growth rate of system status data after the action is implemented, and store the access pattern metadata generated by the analysis.

2. The data cache management method based on big data according to claim 1, characterized in that: The collecting of access pattern metadata and the construction of data cache state space and action space include: Configure an Apache Flink cluster, define the Flink data stream time step window W, and aggregate and collect user access logs and system status data. The user access log data includes data ID, time stamp, user ID and context information; The context information includes the type of data, the business scenario of the data, and the type of user; The system status data includes system load, cache hit rate and cache capacity of each layer; Group user access log data by user ID, record the data ID accessed by each user in the time step window W, extract the context signal data of each recorded data ID, convert the context signal data of each data ID accessed by each user and each recorded data ID in the time step window W into text, use it as the input of the pre-trained BERT model, and output the user behavior vector and data vector; Calculate the number of accesses to the data ID in the time step window W, calculate the cosine similarity between the user behavior vector and the data vector, and assign a priority score to the data of each scenario based on expert experience; Build a Transformer model, collect historical user access log data, convert it into text, and then train the Transformer model. Then, convert the user access log data into text and input it into the trained Transformer model to predict the data access probability. The number of accesses of the data ID in the time step window W, the cosine similarity between user behavior and data, the data allocation priority score of each scenario, and the predicted access probability are constructed as a feature vector; The system state data is standardized, and the standardized system state data and the characteristic vector are combined into a state space to obtain a state vector; The cache capacity change, cache data migration and switching replacement strategy are defined as three actions in the action space to construct the action space.

3. The data cache management method based on big data according to claim 1, characterized in that: The filtering of the effective action set of the data cache refers to calculating the access probability threshold based on the system load of the current time step, setting filtering rules for effective actions, and filtering the effective action set.

4. The data cache management method based on big data according to claim 1, characterized in that: The calculating of the context feature weights includes: Use the pre-trained Word2Vec model to convert user access log data into continuous feature vectors to obtain context features; Normalize the state space data, calculate the business scenario priority score, use the ratio method to calculate the user type request ratio, and use the maximum normalization method to calculate the priority weight and type ratio weight based on the user type request ratio and business scenario priority score. Calculate the Pearson correlation coefficient between the proportion of user type requests, business scenario priority score, and cache hit rate changes. ,Based on the priority weight and type proportion weight, the interaction term is introduced to calculate the context feature weight.

5. The data cache management method based on big data according to claim 1, characterized in that: The construction of the CO-IAM model to predict data caching actions includes: Collect historical state data to perform initial training on the CO-IAM model and input the state vector into the trained CO-IAM model; The self-attention layer is used to extract the dependency of the state vector using the self-attention mechanism and perform weighted summation to obtain an enhanced state; The multi-layer perceptron (MLP) is used to map the enhanced state to the action space and generate logits; The invalid action mask generation unit is used to generate a mask vector based on the valid action set, perform a dot product operation on the mask vector of each action and the generated logits, and output the result as the initial action probability through the softmax function; The incremental data processing unit is used to add access pattern metadata of the newly added time step; The composite objective optimization unit is configured to calculate a basic reward using a weighted sum method based on a cache hit rate and a system load change in the normalized state space data, and adjust the basic reward in combination with a context weight and an action priority; Define the value network, use temporal difference (TD) estimation to calculate the advantage value, use the Actor-Critic algorithm to calculate the actor loss and critic loss based on the advantage value and the enhanced state, and take the weighted sum of the actor loss and the critic loss as the composite loss; With the goal of minimizing the composite loss, the gradient descent method is used to update the policy parameters using the actor loss and the value network parameters using the critic loss. The context weight adjustment unit is used to adjust the initial action probability using the context weight and output the final data cache action probability; The distributed training and reasoning framework allocates nodes for model training and reasoning based on dynamic load scheduling and performs parallel computing.

6. The data cache management method based on big data according to claim 1, characterized in that: The data caching action based on the CO-IAM model prediction result is implemented, including: Based on the predicted action probability of the CO-IAM model, if the action with the maximum predicted probability is the migration action, the data migration target and cost are calculated, and the data migration is performed; If the maximum action prediction probability is the adjusted capacity, calculate the new memory cache capacity and compare whether the new memory cache capacity is greater than the maximum memory cache capacity. If the new memory cache capacity is less than or equal to the maximum memory cache capacity, cache the newly added data. If the new memory cache capacity is larger than the maximum memory cache capacity, hot data will be allocated first until the memory cache is exhausted, and the remaining allocatable capacity will be incrementally allocated to the distributed cache; If the maximum action prediction probability is the switch replacement strategy, calculate the data replacement priority and first delete the data with the lowest priority score.

7. The data cache management method based on big data according to claim 1, characterized in that: The construction of a visual interface to display the growth rate of system status data after the action is implemented and to store the access pattern metadata generated by the analysis includes: Calculate the relative growth rate of system status data after the action is executed, and use React.js to build a visual interface for display; The access pattern metadata generated by the analysis is stored in the database, and security access measures are set. The database backs up the access pattern metadata to the cloud and regularly performs integrity checks on the stored data and backup data. After the test is completed, an integrity test record is generated and stored synchronously in the database.

8. A data cache management system based on big data, based on the data cache management method based on big data according to any one of claims 1 to 7, characterized in that: include, The collection and construction module is used to collect access pattern metadata, including user access logs and system status data, to build the data cache state space and action space, and to filter the effective action set of the data cache; The calculation and prediction module is used to calculate the context feature weights based on the state space and build the CO-IAM model to predict data caching actions; Implementation module, used to implement data caching actions based on the prediction results of the CO-IAM model; The visualization storage module is used to build a visualization interface to display the growth rate of system status data after the action is implemented, and to store the access pattern metadata generated by the analysis.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the data cache management method based on big data according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data cache management method based on big data according to any one of claims 1 to 7 are implemented.

Images