Data migration and cache management method and system for cloud service environment

Abstract

The invention belongs to the technical field of data management, and discloses a data migration and cache management method and system for a cloud service environment. The method comprises the following steps: constructing a storage node analysis model and a data migration strategy generation model; collecting real-time operation and memory information of the storage nodes; performing storage node analysis by using the storage node analysis model; generating a data migration strategy by using a data migration strategy generation model; according to the real-time data migration strategy, a plurality of emigration storage nodes and a plurality of immigration storage nodes in the distributed network are confirmed, data migration and cache management are carried out, and real-time transaction data are generated; and performing consensus on the real-time transaction data based on a plurality of consensus nodes in the distributed network. According to the method, the problems of low efficiency, low resource utilization rate and low reliability and integrity in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data management, and in particular relates to a data migration and cache management method and system for a cloud service environment. Background Art

[0002] In today's information technology era, cloud computing has become an important part of enterprise IT infrastructure. With the surge in data volume, how to efficiently and reliably manage data in a cloud service environment has become a key challenge. Migrating and caching data in a cloud service environment, reducing access latency, and improving the overall performance of the cloud service environment are important measures for data management in a cloud service environment.

[0003] Existing data migration and cache management technologies have the following defects: 1) Inefficiency: Traditional data migration and cache management often rely on manual operations, which is not only inefficient but also prone to errors. In addition, manual management is almost unfeasible in environments with large amounts of data and many storage nodes. 2) Low resource utilization: Many existing data migration strategies are static and cannot be dynamically adjusted according to real-time system load and performance indicators, resulting in low resource utilization and slow cloud service response. 3) Low reliability and integrity: Traditional data migration and cache management fail to provide effective data recovery and retransmission mechanisms, resulting in data loss, low reliability, lack of real-time monitoring and verification of the data migration process, inability to promptly detect and handle data migration failures, and low integrity. Summary of the invention

[0004] In order to solve the problems of low efficiency, low resource utilization, and low reliability and integrity in the prior art, the present invention aims to provide a method and system for data migration and cache management in a cloud service environment.

[0005] The technical solution adopted by the present invention is: A data migration and cache management method for a cloud service environment includes the following steps: In the cloud service environment, artificial intelligence algorithms are used to build storage node analysis models and data migration strategy generation models; Collecting real-time operation and memory information of several storage nodes in the distributed network of the cloud service environment, and performing preprocessing to obtain some preprocessed real-time operation and memory information; Use the storage node analysis model to perform storage node analysis on some pre-processed real-time operation and memory information to obtain real-time storage node analysis results of the cloud service environment; Use the data migration strategy generation model to generate data migration strategies based on the real-time storage node analysis results to obtain the real-time data migration strategy of the cloud service environment; According to the real-time data migration strategy, identify several storage nodes to be migrated out and several storage nodes to be migrated in in the distributed network, perform data migration and cache management, and generate real-time transaction data; Based on several consensus nodes in the distributed network, a consensus algorithm is used to reach consensus on real-time transaction data. If the consensus is successful, data migration and cache management are terminated.

[0006] Furthermore, in the cloud service environment, an artificial intelligence algorithm is used to construct a storage node analysis model and a data migration strategy generation model, including the following steps: In the cloud service environment, historical operation and memory information of several storage nodes are collected and preprocessed to obtain some preprocessed historical operation and memory information; Based on some pre-processed historical operation and memory information, a deep learning algorithm is used to build a storage node analysis model and generate some historical storage node analysis results; Based on the analysis results of several historical storage nodes, a data migration strategy generation model is constructed using a reinforcement learning algorithm, and several historical data migration strategy generation experiences are generated.

[0007] Furthermore, the storage node analysis model is constructed based on the RF-Attention-DBN algorithm, and the storage node analysis model includes a key feature screening module constructed based on the RF algorithm, an attention weight module constructed based on the Attention mechanism, and a storage node analysis module constructed based on the DBN algorithm, which are connected in sequence.

[0008] Furthermore, based on some pre-processed historical operation and memory information, a deep learning algorithm is used to build a storage node analysis model and generate some historical storage node analysis results, including the following steps: Use the RF-Attention-DBN algorithm to build an initial storage node analysis model; the initial storage node analysis model includes an initial key feature screening module, an initial attention weight module, and an initial storage node analysis module; Converting a number of pre-processed historical operations and memory information of the distributed network collected at the same time point into a historical data matrix to obtain a number of historical data matrices; According to several historical data matrices, the initial key feature screening module is trained to obtain a final key feature screening module, and several key feature indicators and several historical key features of each historical data matrix are obtained; According to several historical key features of all historical data matrices, the initial attention weight module is trained to obtain the final attention weight module, and the historical weighted fusion features of several historical data matrices are obtained; According to several historical weighted fusion features, the initial storage node analysis module is trained to obtain the final storage node analysis module, and several historical storage node analysis results are generated; The final key feature screening module, the final attention weight module and the final storage node analysis module are integrated to obtain the final storage node analysis model.

[0009] Furthermore, the data migration strategy generation model is constructed based on the IFWA-PPO algorithm, and the data migration strategy generation model includes a model parameter optimization module constructed based on the IFWA algorithm and a data migration strategy generation module constructed based on the PPO algorithm, which are connected in sequence. Furthermore, according to the analysis results of several historical storage nodes, a data migration strategy generation model is constructed using a reinforcement learning algorithm, and several historical data migration strategy generation experiences are generated, including the following steps: Use the IFWA algorithm to build a model parameter optimization module, and use the model parameter optimization module to iteratively optimize the initial network parameters of the policy network to obtain the optimal initial network parameters; Based on the optimal initial network parameters, a policy network is constructed, the experience replay mechanism is used to initialize the experience replay pool, and the agent is defined; Define the state space, action space and reward function of the agent, and use the PPO algorithm to build the initial data migration strategy generation module based on the strategy network, experience replay pool and agent; According to the analysis results of several historical storage nodes, the initial data migration strategy generation module is trained to obtain the final data migration strategy generation module, and several historical data migration strategy generation experiences are generated; The model parameter optimization module and the final data migration strategy generation module are integrated to obtain the data migration strategy generation model, and several historical data migration strategy generation experiences are stored in the experience replay pool.

[0010] Furthermore, using the storage node analysis model, a storage node analysis is performed on a number of pre-processed real-time operation and memory information to obtain a real-time storage node analysis result of the cloud service environment, including the following steps: Convert several pre-processed real-time operation and memory information collected at the same time point into a real-time data matrix, and input the real-time data matrix into the storage node analysis model; According to several key feature indicators, the key feature screening module of the storage node analysis model is used to screen the key features of the real-time data matrix to obtain several corresponding real-time key features; According to the preset attention weight value, the attention weight module of the storage node analysis model is used to perform weighted fusion on several real-time key features to obtain real-time weighted fusion features; The storage node analysis module of the storage node analysis model is used to perform storage node analysis on the real-time weighted fusion features to obtain real-time storage node analysis results of the cloud service environment.

[0011] Furthermore, using the data migration strategy generation model, the data migration strategy is generated for the real-time storage node analysis results to obtain the real-time data migration strategy of the cloud service environment, including the following steps: Decomposing the real-time storage node analysis results to obtain a number of real-time storage node states, and updating the state space of the data migration strategy generation model according to the real-time storage node states to obtain an updated state space; Randomly extract several historical data migration strategy generation experiences from the experience replay pool, and update the action space of the data migration strategy generation model based on the several historical data migration strategy generation experiences to obtain an updated action space; The intelligent agent using the data migration strategy generation model controls the policy network to generate the probability distribution of all possible data migration actions in the updated action space corresponding to each real-time storage node state in the updated state space based on the reward function; The possible data migration action with the highest probability distribution in the updated action space is taken as the execution data migration action of the real-time storage node state, and the execution data migration actions of all real-time storage node states in the updated state space are integrated to obtain the real-time data migration strategy of the cloud service environment.

[0012] Further, according to the real-time data migration strategy, a number of outgoing storage nodes and a number of incoming storage nodes in the distributed network are identified, data migration and cache management are performed, and real-time transaction data is generated, including the following steps: According to the real-time data migration strategy, identify several storage nodes to be migrated out and several storage nodes to be migrated in in the distributed network; Confirm all data blocks that have been migrated out of the storage node, and obtain the real-time migration score of each data block based on the real-time migration operation and memory information of the migrated storage node; According to the real-time migration score, the real-time migration data of the storage node to be migrated is confirmed, and according to the real-time data migration strategy, the real-time migration data is published to the distributed network, and a real-time migration record is generated; Confirm the real-time free storage resources of the migrated storage node, cache the real-time migrated data in the distributed network according to the real-time free storage resources, and generate real-time cache records; Repeat the above steps to obtain real-time migration records of several outgoing storage nodes and real-time cache records of several incoming storage nodes; A number of real-time migration records and a number of real-time cache records are cancelled. If there are any missing real-time migration records or erroneous real-time cache records, data is re-migrated for the missing real-time migration records and data is restored for the erroneous real-time cache records. Based on all real-time migration records and all real-time cache records, the real-time transaction data is generated using the smart contract of the distributed network, and the real-time transaction data is published to the distributed network.

[0013] A data migration and cache management system for a cloud service environment is used to implement a data migration and cache management method. The system includes a model building unit, an information collection unit, a storage node analysis unit, a data migration strategy generation unit, a data migration and cache management unit, and a transaction data consensus unit that are connected in sequence.

[0014] The beneficial effects of the present invention are: The present invention provides a data migration and cache management method and system for a cloud service environment. By adopting an artificial intelligence algorithm to automate the data migration and cache management process, manual intervention is significantly reduced, and the efficiency of data migration and cache management is greatly improved, especially when processing large-scale data and complex distributed storage networks. The constructed storage node analysis model can realize fine-grained monitoring of the node status, accurately capture the real-time performance and load of the node, thereby improving the accuracy of data migration and cache decisions. The constructed data migration strategy generation model dynamically adjusts the data migration strategy according to the real-time system load and performance indicators, ensuring that storage resources are reasonably allocated and efficiently utilized, thereby improving the response speed and overall performance of cloud services. The constructed migration and cache record cancellation mechanism realizes an effective data recovery and re-migration mechanism, timely discovers and handles errors and failures in data migration, and ensures the reliability of migration and the integrity of data.

[0015] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of the data migration and cache management method in the cloud service environment in the present invention.

[0017] Figure 2 It is a structural block diagram of the data migration and cache management system in the cloud service environment in the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.

[0019] Embodiment 1: like Figure 1 As shown, this embodiment provides a data migration and cache management method for a cloud service environment, including the following steps: S1: In the cloud service environment, use artificial intelligence algorithms to build a storage node analysis model and a data migration strategy generation model, including the following steps: S1-1: In a cloud service environment, historical operation and memory information of several storage nodes is collected and preprocessed to obtain some preprocessed historical operation and memory information; Operation and memory information includes operation monitoring information and memory monitoring information of storage nodes. Operation monitoring information includes: Basic system information: operating system type and version, CPU model, number of cores and frequency, memory size, type and speed, hard disk type, capacity and rotation speed, network interface type and speed; Process and thread information: the number of running processes and threads, the CPU and memory operation of processes and threads, and the status and performance of storage-related processes; Error and log information: system error logs, storage-related errors and warnings, network errors, and packet loss; Cloud service status: status of cloud storage services, status of backup and recovery tasks, high availability and failover status; Temperature and power consumption: CPU and memory temperature, hard disk temperature, system power consumption; Hardware health: self-monitoring of hard disks, analysis and reporting program status, health of memory modules, early warning information (such as battery backup unit status); Memory status monitoring information includes: Storage performance indicators: read and write speed, latency, data transmission error rate, storage utilization (used / total capacity), storage pool redundancy, and fault domain distribution; Memory usage: total memory size, used memory size, free memory size, cache usage, memory page swap activity; System load: CPU usage, memory usage, hard disk I / O usage, network bandwidth usage, system load average; Preprocessing includes data cleaning, Gaussian denoising, and normalization, which improves the quality of samples and provides sample support for subsequent model construction; S1-2: Based on some pre-processed historical operation and memory information, a deep learning algorithm is used to build a storage node analysis model and generate some historical storage node analysis results; The storage node analysis model is built based on the Random Forest (RF)-Attention-Deep Belief Nets (DBN) algorithm, and the storage node analysis model includes a key feature screening module built based on the RF algorithm, an attention weight module built based on the Attention mechanism, and a storage node analysis module built based on the DBN algorithm, which are connected in sequence; The key feature screening module screens the features of the matrix composed of running and memory information through the Classification And Regression Tree (CART) tree to obtain the key score of each feature. The key score is used to evaluate the key degree of the feature. Key feature extraction based on the feature key score helps to understand which factors have the greatest impact on the performance of the storage node. It can process a large number of features and select the most stable and discriminative key features, reducing the amount of data that the analysis model needs to process, improving computational efficiency, avoiding model overfitting, and making the model have better prediction performance on unknown data; reducing data dimensions and improving the interpretability and prediction ability of the model; the attention weight module weights important key features through attention weight values ​​to increase attention to the features of interest. By assigning appropriate weights to key features, the model can more accurately predict the performance and storage status of storage nodes; the storage node analysis module mines the deep relationship between weighted fusion features and storage node analysis labels through deep structures, improving the accuracy of model predictions, having a certain degree of robustness to noise and missing data, and is suitable for data analysis in actual environments; Based on some pre-processed historical operation and memory information, a deep learning algorithm is used to build a storage node analysis model and generate some historical storage node analysis results, including the following steps: S1-2-1: Use the RF-Attention-DBN algorithm to build an initial storage node analysis model; the initial storage node analysis model includes an initial key feature screening module, an initial attention weight module, and an initial storage node analysis module; S1-2-2: converting a number of pre-processed historical operations and memory information of the distributed network collected at the same time point into a historical data matrix to obtain a number of historical data matrices; S1-2-3: According to several historical data matrices, the initial key feature screening module is trained to obtain the final key feature screening module, and several key feature indicators and several historical key features of each historical data matrix are obtained, including the following steps: S1-2-3-1: Use the initial key feature screening module to extract the historical data matrix features of the historical data matrix, divide the historical data matrix features into M candidate features, and obtain the feature contribution of the M candidate features; The formula is:

[0020] In the formula, For the Feature contribution of candidate features; For the The candidate features are in the random forest The feature contribution of each tree; is the CART tree indicator; is an alternative characteristic indicator; is the total number of CARTs;

[0021] In the formula, CART tree node for random forest m ,node and nodes The Gini index of CART tree node m Medium Category The proportion of is the total number of categories; m , , is the node indicator; is the category indicator; S1-2-3-2: normalize the feature contributions of the M candidate features to obtain a number of corresponding normalized feature contributions; The formula is:

[0022] In the formula, It is the contribution of the feature after normalization; J is the total number of candidate features; S1-2-3-3: Generate feature selection standard values ​​for M candidate features based on the normalized feature contribution; The formula is:

[0023] In the formula, For the Feature selection criterion values ​​for candidate features; For the The normalized feature contribution of the candidate features; is an alternative characteristic indicator; S1-2-3-4: Use the feature selection standard value as the feature key score to obtain M feature key scores, sort the M candidate features in descending order according to the feature key scores, and select the top The alternative feature indicators of the candidate features are used as key feature indicators; S1-2-3-5: According to Key feature indicators are used to screen the key features of the historical data matrix and obtain several historical key features of the historical data matrix; S1-2-3-6: traverse all historical data matrices, train the initial key feature screening module, obtain the final key feature screening module, and obtain several historical key features of each historical data matrix; S1-2-4: According to several historical key features of all historical data matrices, the initial attention weight module is trained to obtain the final attention weight module, and the historical weighted fusion features of several historical data matrices are obtained; S1-2-5: Based on several historical weighted fusion features, the initial storage node analysis module is trained to obtain the final storage node analysis module, and several historical storage node analysis results are generated; S1-2-6: Integrate the final key feature screening module, the final attention weight module and the final storage node analysis module to obtain the final storage node analysis model; S1-3: Based on the analysis results of several historical storage nodes, a data migration strategy generation model is constructed using a reinforcement learning algorithm, and several historical data migration strategy generation experiences are generated; The data migration strategy generation model is constructed based on the Improved Fireworks Optimization Algorithm (IFWA)-Proximal Policy Optimization (PPO) algorithm, and the data migration strategy generation model includes a model parameter optimization module constructed based on the IFWA algorithm and a data migration strategy generation module constructed based on the PPO algorithm, which are connected in sequence; The IFWA algorithm used in the model parameter optimization module is used to optimize the initial parameters of the policy network. By simulating the process of fireworks explosion, the optimal parameter combination is searched in the parameter space, which can jump out of the local optimum and find the global optimal solution, thereby providing better initial parameters for the policy network. By optimizing the initial parameters, the training efficiency and policy quality of the subsequent policy network can be improved. The policy network is a neural network used to predict the action that should be taken in a given state. The experience replay pool is used to store the experience of the agent (state, action, reward, next state) so that samples can be reused during training, which can reduce the correlation between experiences and improve the stability of learning. The agent is a decision-making entity that achieves a certain goal by taking actions in the environment. The data migration strategy generation module maximizes long-term rewards by iteratively updating the parameters of the policy network, which improves the stability of policy learning and reduces the fluctuation of policy performance. Through continuous learning and adjustment, the policy network can adapt to different data migration scenarios and needs and generate a highly adaptable migration strategy. Based on the analysis results of several historical storage nodes, a data migration strategy generation model is constructed using a reinforcement learning algorithm, and several historical data migration strategy generation experiences are generated, including the following steps: S1-3-1: Use the IFWA algorithm to build a model parameter optimization module, and use the model parameter optimization module to iteratively optimize the initial network parameters of the policy network to obtain the optimal initial network parameters, including the following steps: S1-3-1-1: Use the IFWA algorithm to build a model parameter optimization module; S1-3-1-2: Use the model parameter optimization module to minimize the network error as the optimization goal, and set the individual coding vector format and IFWA population parameters according to the initial network parameters; S1-3-1-3: Set the fitness function of the IFWA algorithm according to the optimization goal; The formula is:

[0024] In the formula, is the fitness function; is the mean square error; S1-3-1-4: Initialize using the Circle chaotic mapping sequence according to the individual coding vector format and IFWA population parameters to generate several initial IFWA individuals of the initial IFWA population; The formula is:

[0025] In the formula, is the initial IFWA individual of Circle chaos mapping; is the randomly generated initial IFWA individual; It is the IFWA individual indicator; S1-3-1-5: According to the fitness function, the explosion radius, number of sparks and fitness value of the initial IFWA individual are obtained; The formula is:

[0026] In the formula, For the initial IFWA individuals The number of sparks; is a numerical constant; is the maximum fitness value in the initialized IFWA population; For the initial IFWA individuals The fitness value of is an infinitesimal constant; is the convergence factor; is a positive real number that is not 0;

[0027] In the formula, For the initial IFWA individuals Explosion radius; Adjusted constant for explosion radius; is the minimum fitness value in the initialized IFWA population; is the total number of IFWA individuals;

[0028] In the formula, is the convergence factor; tanh(.) is the hyperbolic tangent function; is the iteration indicator; is the maximum number of iterations; a max , a min are the maximum and minimum values ​​of the convergence factor, respectively; λ is the deceleration rate parameter, is the decreasing cycle parameter, λ =-2 π , = π ; The number of sparks determines the number of sub-fireworks produced after each firework explodes, and the explosion radius determines the distribution range of the sparks produced after the firework explodes in the solution space. In the early stage of iteration, aThe larger the value of , the smaller the number of sparks of IFWA individuals and the larger the explosion radius, which helps to reduce the computational burden and is more widely distributed, which helps to explore more solution spaces. In the later stages of iteration, a smaller explosion radius helps to perform a fine search in a local area, and a larger number of sparks helps to increase the diversity of the search. S1-3-1-6: Perform fireworks explosion according to the explosion radius, the number of sparks and the fitness value to obtain a number of updated IFWA individuals of the updated IFWA population; The formula is:

[0029] In the formula, For updated IFWA individuals; A random number between -1 and 1; S1-3-1-7: Use the Gaussian mutation algorithm to perform Gaussian mutation on the initialized IFWA population to generate several Gaussian mutated IFWA individuals of the Gaussian mutated IFWA population; The formula is:

[0030] In the formula, is the IFWA individual of Gaussian variation; is a Gaussian distributed random number with mean and variance both 1; S1-3-1-8: Use the dynamic reverse learning algorithm to perform dynamic reverse learning on the initialized IFWA population to obtain several reverse IFWA individuals of the reverse IFWA population; The formula is:

[0031] In the formula, For reverse IFWA individuals; γ is the decreasing inertia coefficient; L max , L min are the maximum and minimum values ​​of the vector space respectively; S1-3-1-9: Obtain the fitness value of each updated IFWA individual, Gaussian mutated IFWA individual and reverse IFWA individual, and take the IFWA individual with the minimum fitness value as the optimal individual; S1-3-1-10: If the number of iterations reaches the maximum number of iterations or the fitness value of the optimal individual meets the requirements, the individual encoding vector of the optimal individual is decoded to obtain the optimal initial network parameters; S1-3-2: Based on the optimal initial network parameters, build a policy network, use the experience replay mechanism, initialize the experience replay pool, and define the agent; S1-3-3: Define the state space, action space and reward function of the agent, and use the PPO algorithm to build the initial data migration strategy generation module based on the strategy network, experience replay pool and agent; S1-3-4: Based on the analysis results of several historical storage nodes, the initial data migration strategy generation module is trained to obtain the final data migration strategy generation module, and several historical data migration strategy generation experiences are generated; the historical storage node analysis results are used as training data, and the policy network is trained through the PPO algorithm. During the training process, the intelligent agent learns the best data migration strategy by interacting with the environment; through training, the intelligent agent accumulates data migration experience, which can be used for future decision-making; S1-3-5: Integrate the model parameter optimization module and the final data migration strategy generation module to obtain a data migration strategy generation model, and store several historical data migration strategy generation experiences into the experience playback pool; S2: collect real-time operation and memory information of several storage nodes in the distributed network of the cloud service environment, and perform preprocessing to obtain some preprocessed real-time operation and memory information; S3: Use the storage node analysis model to perform storage node analysis on some pre-processed real-time operation and memory information to obtain the real-time storage node analysis results of the cloud service environment, including the following steps: S3-1: converting a number of pre-processed real-time operation and memory information collected at the same time point into a real-time data matrix, and inputting the real-time data matrix into the storage node analysis model; The real-time data matrix is ​​usually a multidimensional array, in which each row represents the information of a storage node and each column represents a specific feature indicator. Converting the data into a matrix format provides a unified input format for subsequent model processing; S3-2: Based on several key feature indicators, use the key feature screening module of the storage node analysis model to screen the key features of the real-time data matrix and obtain several corresponding real-time key features. Focusing on the most important features can more quickly identify the status and potential problems of storage nodes. S3-3: According to the preset attention weight value, the attention weight module of the storage node analysis model is used to perform weighted fusion on several real-time key features to obtain real-time weighted fusion features; the attention mechanism can set weights according to the importance of the features, so that important features have a greater influence in the analysis; S3-4: Use the storage node analysis module of the storage node analysis model to perform storage node analysis on the real-time weighted fusion features to obtain real-time storage node analysis results of the cloud service environment; the weighted fusion features can more accurately express the real-time status of the operation of the storage node and the memory; The storage node analysis results include: storage node performance indicator analysis results, resource utilization analysis results, node health status analysis results, resource demand prediction results, and fault prediction results; S4: Using the data migration strategy generation model, a data migration strategy is generated for the real-time storage node analysis results to obtain a real-time data migration strategy for the cloud service environment, including the following steps: S4-1: Decompose the real-time storage node analysis results to obtain several real-time storage node states, and update the state space of the data migration strategy generation model based on the several real-time storage node states to obtain an updated state space, ensuring that the model can take into account the latest storage node status. The model can dynamically adjust the strategy according to the current storage node status to improve the adaptability of the strategy. The updated state space more accurately reflects the real-time situation of the cloud service environment. The storage node status includes the storage node numerical status, storage node classification status and storage node Boolean status; Storage node numerical status includes: CPU usage (0% - 100%), memory usage (in MB or GB), disk I / O rate (number of operations per second), network throughput (in Mbps), storage utilization (0% - 100%), read and write operation response time (in milliseconds); Storage node classification status includes: node status (such as normal operation, overload, failure, maintenance, etc.), resource status (such as sufficient, tight, insufficient, etc.), health status (such as good, warning, serious, etc.); Storage node Boolean states include: whether a hardware failure has occurred (yes / no), whether a resource threshold has been reached (yes / no), whether it is in maintenance mode (yes / no); In the data migration strategy generation model, these state values ​​will be used as inputs of the agent, and the agent will choose appropriate actions based on these state values ​​(for example, whether to migrate data, how much data to migrate, which node to migrate to, etc.). In this way, the data migration strategy generation model can learn how to adopt the optimal data migration strategy under different storage node states; S4-2: Randomly extract several historical data migration strategy generation experiences from the experience replay pool, and update the action space of the data migration strategy generation model based on the several historical data migration strategy generation experiences to obtain an updated action space. These historical experiences are used to update the action space of the data migration strategy generation model, thereby introducing prior knowledge based on historical data. The model can learn more effective data migration strategies. The updated action space provides more possible migration actions, increasing the diversity of strategies. S4-3: The intelligent agent of the data migration strategy generation model controls the policy network and generates the probability distribution of all possible data migration actions in the updated action space corresponding to each real-time storage node state in the updated state space based on the reward function. Through the probability distribution, the model can evaluate the possibility of different migration actions and provide a quantitative basis for decision-making. Probability-based decision-making reduces the risk of blindly migrating data. S4-4: taking the possible data migration action with the highest probability distribution in the updated action space as the execution data migration action of the real-time storage node state, integrating the execution data migration actions of all real-time storage node states in the updated state space, and obtaining the real-time data migration strategy of the cloud service environment; Data migration strategy is a set of rules and operations that guide the migration of data between different storage nodes in the cloud service environment. Data migration content includes decisions such as data block selection, storage node selection, migration timing selection, and migration method selection. Data block selection: Determine which data blocks need to be migrated, which is usually based on factors such as the size, access frequency, modification frequency, and importance of the data blocks; Migration storage node selection: Determine which storage node the data should be migrated to, which is usually based on factors such as the target node's storage space, performance, network latency, and load conditions; Migration timing selection: Determine when to start data migration based on factors such as time window, network traffic, node load, etc. Migration method selection: select a specific method for data migration, such as synchronous migration, asynchronous migration, incremental migration, full migration, etc. Based on the above data migration content, the data migration action is obtained: Read operation action: read the data blocks to be migrated from the source storage node; Data transfer action: transfer the read data block to the target storage node through the network; Write operation action: write the data block transmitted to the target node to the target storage medium; Data validation actions: ensuring the integrity and consistency of data during the migration process, which may include checksum or hash verification; Metadata update action: Updates the location information and other metadata of the data block to reflect the new storage location; Source data cleanup action: After confirming that the data has been successfully migrated to the target node, delete the original data blocks from the source node to free up space; Migration log action: records detailed information about data migration, including migration time, data block ID, source node, target node, etc. Error handling actions: When an error occurs during the migration process, execute the corresponding error recovery or retry strategy; Load balancing action: adjust the priority and order of data migration according to the real-time load of the node; S5: According to the real-time data migration strategy, several outgoing storage nodes and several incoming storage nodes in the distributed network are identified, data migration and cache management are performed, and real-time transaction data is generated, including the following steps: S5-1: According to the real-time data migration strategy, identify several storage nodes to be migrated out and several storage nodes to be migrated in in the distributed network; the nodes to be migrated out are usually nodes with high resource utilization or reduced performance, while the nodes to be migrated in are nodes with sufficient resources or better performance. By migrating data, the resource load in the entire network can be balanced. Migrating data to nodes with better performance can improve the data access speed of the cloud service environment; S5-2: confirm all data blocks to be migrated out of the storage node, and obtain the real-time migration score of each data block according to the real-time migration operation and memory information of the migrated storage node. The migration score is usually calculated based on factors such as the importance, access frequency, and size of the data block. Through the score, it can be determined which data blocks need to be migrated the most, and key data can be processed first; S5-3: According to the real-time migration score, confirm the real-time migration data of the migrated storage node, publish the real-time migration data to the distributed network according to the real-time data migration strategy, and generate a real-time migration record, which provides historical information of data migration to facilitate auditing and troubleshooting; S5-4: confirm the real-time idle storage resources of the migrated storage node, cache the real-time migrated data in the distributed network according to the real-time idle storage resources, and generate real-time cache records, and use the idle resources of the migrated node to store the migrated data to optimize the overall storage efficiency, effectively use the idle resources, and reduce waste. Data caching can reduce the delay of data access; S5-5: Repeat the above steps to obtain real-time migration records of several outgoing storage nodes and real-time cache records of several incoming storage nodes; S5-6: Verify several real-time migration records and several real-time cache records. If there are missing real-time migration records or erroneous real-time cache records, re-migrate the data of the missing real-time migration records and recover the data of the erroneous real-time cache records. Check the accuracy of migration and cache records. If there are omissions or errors, correct them to ensure that the migrated and cached data are accurate and maintain data consistency. S5-7: Generate real-time transaction data based on all real-time migration records and all real-time cache records using the smart contract of the distributed network, and publish the real-time transaction data to the distributed network. Through the smart contract, the security of the transaction data is guaranteed, and the publication of the transaction data increases the transparency of the system, facilitating monitoring and trust building; S6: Based on several consensus nodes in the distributed network, a consensus algorithm is used to reach a consensus on real-time transaction data. If the consensus is successful, data migration and cache management are terminated, including the following steps: S6-1: Define the nodes participating in the consensus in the distributed network. These consensus nodes are usually screened nodes with high credibility and computing power; S6-2: The generated real-time transaction data is sent to each consensus node. Each consensus node verifies the real-time transaction data using a predetermined consensus algorithm (such as proof of work, proof of stake, Byzantine fault tolerance algorithm, etc.) to ensure that all nodes reach a consensus on the status of the transaction data and obtain a real-time consensus result; S6-3: If the consensus is successful, it means that most of the consensus nodes in the distributed network agree on the correctness of the transaction data, then the data migration and cache management process is considered complete, and the data migration and cache management are terminated; S6-4: If consensus fails, re-execute the data migration and cache management process.

[0032] Embodiment 2: like Figure 2 As shown, this embodiment provides a data migration and cache management system for a cloud service environment, which is used to implement a data migration and cache management method. The system includes a model building unit, an information collection unit, a storage node analysis unit, a data migration strategy generation unit, a data migration and cache management unit, and a transaction data consensus unit connected in sequence.

[0033] A model building unit, used to build a storage node analysis model and a data migration strategy generation model using an artificial intelligence algorithm in a cloud service environment; An information collection unit, used to collect real-time operation and memory information of a number of storage nodes in a distributed network of a cloud service environment, and perform preprocessing to obtain a number of preprocessed real-time operation and memory information; A storage node analysis unit, used to use a storage node analysis model to perform storage node analysis on a number of pre-processed real-time operation and memory information to obtain a real-time storage node analysis result of a cloud service environment; A data migration strategy generation unit, used to generate a data migration strategy based on the real-time storage node analysis results using a data migration strategy generation model, and obtain a real-time data migration strategy for a cloud service environment; A data migration and cache management unit, which is used to identify a number of storage nodes to be migrated out and a number of storage nodes to be migrated in in a distributed network according to a real-time data migration strategy, perform data migration and cache management, and generate real-time transaction data; The transaction data consensus unit is used to reach consensus on real-time transaction data based on several consensus nodes in the distributed network using a consensus algorithm. If the consensus is successful, data migration and cache management are terminated.

[0034] The present invention provides a data migration and cache management method and system for a cloud service environment. By adopting an artificial intelligence algorithm to automate the data migration and cache management process, manual intervention is significantly reduced, and the efficiency of data migration and cache management is greatly improved, especially when processing large-scale data and complex distributed storage networks. The constructed storage node analysis model can realize fine-grained monitoring of the node status, accurately capture the real-time performance and load of the node, thereby improving the accuracy of data migration and cache decisions. The constructed data migration strategy generation model dynamically adjusts the data migration strategy according to the real-time system load and performance indicators, ensuring that storage resources are reasonably allocated and efficiently utilized, thereby improving the response speed and overall performance of cloud services. The constructed migration and cache record cancellation mechanism realizes an effective data recovery and re-migration mechanism, timely discovers and handles errors and failures in data migration, and ensures the reliability of migration and the integrity of data.

[0035] The present invention is not limited to the above optional implementations, and anyone can derive other various forms of products under the enlightenment of the present invention. The above specific implementations should not be understood as limiting the scope of protection of the present invention. The scope of protection of the present invention should be based on the definition in the claims, and the description can be used to interpret the claims.

Claims

1. A method for data migration and cache management in a cloud service environment, characterized by: The steps include: In the cloud service environment, artificial intelligence algorithms are used to build storage node analysis models and data migration strategy generation models; Collecting real-time operation and memory information of several storage nodes in the distributed network of the cloud service environment, and performing preprocessing to obtain some preprocessed real-time operation and memory information; Use the storage node analysis model to perform storage node analysis on some pre-processed real-time operation and memory information to obtain real-time storage node analysis results of the cloud service environment; Use the data migration strategy generation model to generate data migration strategies based on the real-time storage node analysis results to obtain the real-time data migration strategy of the cloud service environment; According to the real-time data migration strategy, identify several storage nodes to be migrated out and several storage nodes to be migrated in in the distributed network, perform data migration and cache management, and generate real-time transaction data; Based on several consensus nodes in the distributed network, a consensus algorithm is used to reach consensus on real-time transaction data. If the consensus is successful, data migration and cache management are terminated.

2. A method for data migration and cache management in a cloud service environment according to claim 1, characterized in that: In the cloud service environment, artificial intelligence algorithms are used to build storage node analysis models and data migration strategy generation models, including the following steps: In the cloud service environment, historical operation and memory information of several storage nodes are collected and preprocessed to obtain some preprocessed historical operation and memory information; Based on some pre-processed historical operation and memory information, a deep learning algorithm is used to build a storage node analysis model and generate some historical storage node analysis results; Based on the analysis results of several historical storage nodes, a data migration strategy generation model is constructed using a reinforcement learning algorithm, and several historical data migration strategy generation experiences are generated.

3. A method for data migration and cache management in a cloud service environment according to claim 2, characterized in that: The storage node analysis model is constructed based on the RF-Attention-DBN algorithm, and the storage node analysis model includes a key feature screening module constructed based on the RF algorithm, an attention weight module constructed based on the Attention mechanism, and a storage node analysis module constructed based on the DBN algorithm, which are connected in sequence.

4. A method for data migration and cache management in a cloud service environment according to claim 3, characterized in that: Based on some pre-processed historical operation and memory information, a deep learning algorithm is used to build a storage node analysis model and generate some historical storage node analysis results, including the following steps: Using the RF-Attention-DBN algorithm, an initial storage node analysis model is constructed; the initial storage node analysis model includes an initial key feature screening module, an initial attention weight module, and an initial storage node analysis module; Converting a number of pre-processed historical operations and memory information of the distributed network collected at the same time point into a historical data matrix to obtain a number of historical data matrices; According to several historical data matrices, the initial key feature screening module is trained to obtain a final key feature screening module, and several key feature indicators and several historical key features of each historical data matrix are obtained; According to several historical key features of all historical data matrices, the initial attention weight module is trained to obtain the final attention weight module, and the historical weighted fusion features of several historical data matrices are obtained; According to several historical weighted fusion features, the initial storage node analysis module is trained to obtain the final storage node analysis module, and several historical storage node analysis results are generated; The final key feature screening module, the final attention weight module and the final storage node analysis module are integrated to obtain the final storage node analysis model.

5. A method for data migration and cache management in a cloud service environment according to claim 4, characterized in that: The data migration strategy generation model is constructed based on the IFWA-PPO algorithm, and the data migration strategy generation model includes a model parameter optimization module constructed based on the IFWA algorithm and a data migration strategy generation module constructed based on the PPO algorithm, which are connected in sequence.

6. A method for data migration and cache management in a cloud service environment according to claim 5, characterized in that: Based on the analysis results of several historical storage nodes, a data migration strategy generation model is constructed using a reinforcement learning algorithm, and several historical data migration strategy generation experiences are generated, including the following steps: Use the IFWA algorithm to build a model parameter optimization module, and use the model parameter optimization module to iteratively optimize the initial network parameters of the policy network to obtain the optimal initial network parameters; Based on the optimal initial network parameters, a policy network is constructed, the experience replay mechanism is used to initialize the experience replay pool, and the agent is defined; Define the state space, action space and reward function of the agent, and use the PPO algorithm to build the initial data migration strategy generation module based on the policy network, experience replay pool and agent; According to the analysis results of several historical storage nodes, the initial data migration strategy generation module is trained to obtain the final data migration strategy generation module, and several historical data migration strategy generation experiences are generated; The model parameter optimization module and the final data migration strategy generation module are integrated to obtain the data migration strategy generation model, and several historical data migration strategy generation experiences are stored in the experience replay pool.

7. A method for data migration and cache management in a cloud service environment according to claim 6, characterized in that: Using the storage node analysis model, storage node analysis is performed on some pre-processed real-time operation and memory information to obtain real-time storage node analysis results of the cloud service environment, including the following steps: Convert several pre-processed real-time operation and memory information collected at the same time point into a real-time data matrix, and input the real-time data matrix into the storage node analysis model; According to several key feature indicators, the key feature screening module of the storage node analysis model is used to screen the key features of the real-time data matrix to obtain several corresponding real-time key features; According to the preset attention weight value, the attention weight module of the storage node analysis model is used to perform weighted fusion on several real-time key features to obtain real-time weighted fusion features; The storage node analysis module of the storage node analysis model is used to perform storage node analysis on the real-time weighted fusion features to obtain real-time storage node analysis results of the cloud service environment.

8. A method for data migration and cache management in a cloud service environment according to claim 7, characterized in that: The data migration strategy generation model is used to generate data migration strategies based on the real-time storage node analysis results to obtain the real-time data migration strategy of the cloud service environment, including the following steps: Decomposing the real-time storage node analysis results to obtain a number of real-time storage node states, and updating the state space of the data migration strategy generation model according to the real-time storage node states to obtain an updated state space; Randomly extract several historical data migration strategy generation experiences from the experience replay pool, and update the action space of the data migration strategy generation model based on the several historical data migration strategy generation experiences to obtain an updated action space; The intelligent agent of the data migration strategy generation model controls the policy network to generate the probability distribution of all possible data migration actions in the updated action space corresponding to each real-time storage node state in the updated state space based on the reward function; The possible data migration action with the highest probability distribution in the updated action space is taken as the execution data migration action of the real-time storage node state, and the execution data migration actions of all real-time storage node states in the updated state space are integrated to obtain the real-time data migration strategy of the cloud service environment.

9. A method for data migration and cache management in a cloud service environment according to claim 8, characterized in that: According to the real-time data migration strategy, several outgoing storage nodes and several incoming storage nodes in the distributed network are identified, data migration and cache management are performed, and real-time transaction data is generated, including the following steps: According to the real-time data migration strategy, identify several storage nodes to be migrated out and several storage nodes to be migrated in in the distributed network; Confirm all data blocks that have been migrated out of the storage node, and obtain the real-time migration score of each data block based on the real-time migration operation and memory information of the migrated storage node; According to the real-time migration score, the real-time migration data of the storage node to be migrated is confirmed, and according to the real-time data migration strategy, the real-time migration data is published to the distributed network, and a real-time migration record is generated; Confirm the real-time free storage resources of the migrated storage node, cache the real-time migrated data in the distributed network according to the real-time free storage resources, and generate real-time cache records; Repeat the above steps to obtain real-time migration records of several outgoing storage nodes and real-time cache records of several incoming storage nodes; A number of real-time migration records and a number of real-time cache records are cancelled. If there are any missing real-time migration records or erroneous real-time cache records, data is re-migrated for the missing real-time migration records and data is restored for the erroneous real-time cache records. Based on all real-time migration records and all real-time cache records, the real-time transaction data is generated using the smart contract of the distributed network, and the real-time transaction data is published to the distributed network.

10. A data migration and cache management system in a cloud service environment, used to implement the data migration and cache management method according to any one of claims 1 to 9, characterized in that: The system comprises a model building unit, an information collection unit, a storage node analysis unit, a data migration strategy generation unit, a data migration and cache management unit and a transaction data consensus unit which are connected in sequence.

Images