A data migration method, device and equipment of a hybrid storage architecture and a medium

Abstract

The application discloses a data migration method and device of a hybrid storage architecture, equipment and a medium, and relates to the technical field of data storage. The method comprises the following steps: acquiring multi-source heterogeneous data under the hybrid storage architecture, performing feature extraction on the multi-source heterogeneous data, and generating a data heat score; taking the data and storage system state information as inputs, and generating a data migration strategy by using a multi-agent reinforcement learning model trained based on a federal learning framework; simulating future data access patterns by using a generative artificial intelligence model, generating a forward-looking migration instruction, and performing a data migration operation between a first storage medium and a second storage medium according to the data migration strategy; wherein the unit storage cost of the first storage medium is higher than that of the second storage medium, and the data access delay of the first storage medium is lower than that of the second storage medium. Through the technical scheme of the application, intelligent, low-delay and high-consistency data dynamic migration and resource optimization configuration between local high-speed storage and object storage can be realized.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a data migration method, apparatus, device, and medium for a hybrid storage architecture. Background Technology

[0002] In the era of explosive data growth driven by artificial intelligence, data has become a core resource for development across various fields. Clustered storage technology has consequently become a focal point of attention. Enterprise storage providers urgently need to achieve efficient and secure storage of massive amounts of data, balancing high storage value with low cost to meet the development needs of fields such as autonomous driving, smart government, and AI large-scale model iteration. Providing high-quality massive data storage solutions has also become crucial for market competition. Currently, the development of modern enterprise storage systems faces numerous challenges. Not only is there a significant contradiction between storage cost and performance, but there are also issues such as insufficient prediction of multi-dimensional data popularity features, and a lack of unified strategies for collaboration in multi-node hybrid storage systems, which can easily lead to resource waste and data inconsistency. Existing data migration technologies, as an important support for storage system optimization, still have significant limitations. Their mainstream migration strategies rely on static rules such as time thresholds and access frequency, failing to dynamically adapt to business changes. Some dynamic migration solutions can only achieve single-dimensional feature analysis such as access frequency, making it difficult to conduct intelligent analysis in conjunction with data type and actual business scenarios. Furthermore, migration under multi-node collaborative training modes can easily expose raw data, posing significant compliance risks. The challenges in storage system development and the limitations of data migration technology mutually constrain each other, making it difficult to meet the storage and management needs of massive enterprise-level data. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a data migration method, apparatus, device, and medium with a hybrid storage architecture, capable of integrating multi-dimensional perception, dynamic intelligent decision-making, and privacy protection. The specific solution is as follows: In a first aspect, this application discloses a data migration method for a hybrid storage architecture, including: Acquire multi-source heterogeneous data under a hybrid storage architecture, extract features from the multi-source heterogeneous data, and generate a data popularity score based on the extracted data features; the data popularity score is a quantitative indicator used to characterize the likelihood of data being accessed. Using data popularity scores and storage system status information as inputs, a data migration strategy is generated using a multi-agent reinforcement learning model trained based on a federated learning framework. Generative artificial intelligence models are used to simulate future data access patterns to generate forward-looking migration instructions. Based on the forward-looking migration instructions and data migration strategies, data migration operations are performed between a first storage medium and a second storage medium. The unit storage cost of the first storage medium is higher than that of the second storage medium, and the data access latency of the first storage medium is lower than that of the second storage medium.

[0004] Optionally, acquire multi-source heterogeneous data under a hybrid storage architecture, and extract features from the multi-source heterogeneous data to generate a data popularity score based on the extracted data features, including: Multi-source heterogeneous data under a hybrid storage architecture can be obtained through application programming interfaces or local file systems; the multi-source heterogeneous data includes visual data, text data and structured data, and supports batch processing and real-time streaming input; Visual data is subjected to feature extraction to generate image semantic features; image semantic features include text content features extracted by calling optical character recognition algorithms to recognize characters and then performing keyword matching on the recognized characters using regular expressions, as well as visual structure features extracted by convolutional neural networks. The text data is vectorized, and the vectorized data is then input into a natural language processing model to generate text semantic features. Metadata parsing and normalization are performed on structured data to generate metadata features; The attention mechanism dynamically assigns weights to image semantic features, text semantic features, and metadata features to generate data popularity scores through weighted feature fusion.

[0005] Optionally, data popularity scores and storage system status information are used as inputs to generate data migration strategies using a multi-agent reinforcement learning model trained based on a federated learning framework, including: Each storage node in the hybrid storage architecture is defined as an agent, and a state space, action space, and reward function are set for each agent. The state space includes data popularity score, local storage remaining capacity, access frequency, and network bandwidth usage. The action space includes data migration, data retention, and data preloading. The reward function calculates the reward value based on the changes in storage cost and access latency after migration. Collect local operational data from each agent and use the Q-learning algorithm to train their respective reinforcement learning models locally to generate local model parameters and model gradient information; The model gradient information uploaded by each agent is obtained through a central coordination server, and homomorphic encryption technology is used to encrypt and aggregate the model gradient information to generate global model parameters and distribute them to each agent; the encryption and aggregation process satisfies the differential privacy protection mechanism. The local model parameters of each agent are updated using global model parameters, so that each agent can generate a data migration strategy based on the updated reinforcement learning model.

[0006] Optionally, during the data migration operation between the first storage medium and the second storage medium, based on forward-looking migration instructions and data migration strategies, the process may also include: When multiple storage nodes simultaneously perform data migration operations on the same data object, conflict arbitration is carried out through a preset multi-node negotiation protocol; the preset multi-node negotiation protocol adopts a two-phase commit mechanism. After conflict arbitration, a data migration operation is performed between the first storage medium and the second storage medium. The data migration strategy is dynamically adjusted according to the changes in the first storage medium and the second storage medium to update the reward function parameters corresponding to the agent.

[0007] Optionally, generative artificial intelligence models can be used to simulate future data access patterns to generate forward-looking migration instructions, including: Historical access logs are used as training data, and a diffusion model is used as a generative artificial intelligence model to learn the distribution of data access patterns. Based on the current business status of the storage system, a diffusion model is used to generate a target number of future data access patterns. The data migration costs and business benefits under each data access mode are evaluated to determine the prediction confidence of the diffusion model output. When the prediction confidence level is greater than a preset threshold, a forward migration instruction is generated to preload the predicted hot data into the first storage medium.

[0008] Optionally, the first storage medium includes a local solid-state drive cache, and the second storage medium includes a public cloud object storage layer; according to forward-looking migration instructions and data migration strategies, data migration operations are performed between the first and second storage media, including: The local SSD cache is managed using the Least Recently Used algorithm to limit the maximum local cache capacity, and predicted hot data is preloaded from the public cloud object storage layer to the local SSD cache based on forward-looking migration instructions. According to the data migration strategy, the remote object storage interface is called to migrate the predicted cold data to the public cloud object storage layer, and the original file copy of the cold data is retained in the public cloud object storage layer.

[0009] Optionally, after performing the data migration operation between the first storage medium and the second storage medium according to the forward migration instructions and data migration strategy, the method further includes: Perform hash verification on the data before and after the migration; If the source hash value before migration matches the target hash value after migration, the migration is marked as successful, and a consistency report is generated. The consistency report includes the migration status, changes in storage costs, and verification results. If the source hash value before migration does not match the target hash value after migration, an error log is recorded and a retransmission mechanism is triggered.

[0010] Secondly, this application discloses a data migration apparatus with a hybrid storage architecture, comprising: The data popularity score generation module is used to acquire multi-source heterogeneous data under a hybrid storage architecture, extract features from the multi-source heterogeneous data, and generate a data popularity score based on the extracted data features; the data popularity score is a quantitative indicator used to characterize the likelihood of data being accessed. The data migration strategy generation module takes data popularity score and storage system status information as input and uses a multi-agent reinforcement learning model trained based on a federated learning framework to generate a data migration strategy. The data migration module is used to simulate future data access patterns using a generative artificial intelligence model to generate forward-looking migration instructions, and to execute data migration operations between the first storage medium and the second storage medium according to the forward-looking migration instructions and the data migration strategy; wherein the unit storage cost of the first storage medium is higher than that of the second storage medium, and the data access latency of the first storage medium is lower than that of the second storage medium.

[0011] Thirdly, this application discloses an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned data migration method of the hybrid storage architecture.

[0012] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned data migration method of the hybrid storage architecture.

[0013] The beneficial effects of this application are as follows: First, by collecting multi-source heterogeneous data and extracting features, the generated data popularity score can accurately quantify data popularity, solving the single-dimensional evaluation defect of traditional methods that only rely on access frequency, and significantly improving the accuracy of hot data identification. Second, by adopting a decision-making mechanism that combines federated learning and multi-agent reinforcement learning, in a multi-node distributed storage scenario, through the federated learning framework, each storage node only shares encrypted model gradients rather than the original data, effectively solving the risk of data privacy leakage in cross-node collaborative training. At the same time, the multi-agent reinforcement learning architecture can dynamically adjust the migration strategy according to the real-time business load, overcoming the defect that traditional static threshold rules cannot adapt to sudden business traffic. Finally, it innovatively introduces a generative artificial intelligence model to simulate future potential access patterns, realizing forward-looking migration: cold data is migrated to low-cost storage media, significantly reducing storage costs while ensuring data accessibility; while predicted hot data is preloaded to high-performance storage media, forming a combination strategy of "low-cost storage + high-performance caching". In this way, the performance advantages of the first storage medium and the cost advantages of the second storage medium are effectively balanced, the sudden access latency is greatly reduced, and the global optimization of storage cost and access performance is achieved while ensuring data consistency.

[0014] Furthermore, the data migration apparatus, device, and storage medium provided in this application for a hybrid storage architecture correspond to the data migration method for the aforementioned hybrid storage architecture and have the same effect. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a flowchart of a data migration method for a hybrid storage architecture disclosed in this application; Figure 2 This is a schematic diagram of a data migration system architecture based on a hybrid storage architecture disclosed in this application; Figure 3 This is a flowchart of a data acquisition module disclosed in this application; Figure 4 This is a flowchart of a strategy decision-making module disclosed in this application; Figure 5 This is a flowchart of a migration execution module disclosed in this application; Figure 6 This is a flowchart of a status verification module disclosed in this application; Figure 7 This is a schematic diagram of a data migration device with a hybrid storage architecture disclosed in this application; Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] We are currently in an era of explosive data growth. With artificial intelligence technologies and large-scale model algorithms flourishing across various industries, data, as a foundational resource, has become a vital asset. Enterprises are increasingly recognizing data as a core competitive advantage, and correspondingly, clustered storage technologies are attracting growing attention. How to efficiently and securely store massive amounts of data, achieving a better integration of high value and low cost to bring greater value to customers, has become a key consideration for all enterprise-level storage providers. In the foreseeable future, data will serve as the foundation of the AI ​​era. Whether it's autonomous driving, smart government, or the iteration of AI large-scale model assistants, these hot areas all require high-quality, massive amounts of data as a foundation for optimization. Providing more efficient massive data storage solutions will undoubtedly become a key competitive advantage in the market.

[0019] Based on the current development status of mainstream enterprise-level storage systems, modern enterprise-level storage systems face the following challenges: (1) Contradiction between storage cost and performance: Local solid-state SSD storage devices provide high-performance storage, but the cost of a large number of applications is high. Using the S3 protocol to place remote storage clusters is cost-controllable, but the access latency is high; (2) Insufficient data heat prediction: Traditional methods rely only on data access frequency to predict hot data, lacking multi-dimensional features such as data type and business scenario; (3) Difficulty in cross-node collaboration: Multi-node hybrid storage systems lack a unified strategy, which can easily lead to resource waste and data inconsistency. Existing data migration technologies also have limitations: (1) Current mainstream data migration strategies still rely on static rules, such as time thresholds and access counts, which cannot dynamically adapt to business changes; (2) Existing dynamic migration solutions usually only focus on a single feature, such as access frequency, and cannot intelligently analyze based on data type and actual business scenario; (3) Using multi-node collaborative training can easily lead to the exposure of raw data, resulting in significant compliance risks.

[0020] The aforementioned issues constrain storage efficiency and scenario adaptability, necessitating breakthroughs with superior technical solutions. Analysis of existing solutions reveals that Solution A achieves data migration between HDDs (Hard Disk Drives) and SSDs (Solid State Drives), but its application scenarios primarily focus on video data, with migration strategies based solely on time thresholds and lacking multi-dimensional data feature analysis. Solution B employs a self-learning strategy, optimizing data migration based on user habits over a certain period and incorporating input and output data into the migration considerations. However, this self-learning process is time-consuming to develop an effective migration strategy, making it difficult to adapt to application scenarios with rapidly changing and massively generated data. Furthermore, its technical design primarily targets memory-level data migration, exhibiting significant limitations in practical applications within massive file storage scenarios.

[0021] To this end, this application provides a data migration solution based on a hybrid storage architecture, which not only achieves deep integration and dynamic adaptation of object storage and local storage, but also achieves an optimal balance between data consistency, storage cost and performance through intelligent resource allocation strategies, providing an efficient, secure and scalable solution for large-scale data management in hybrid cloud storage scenarios.

[0022] This invention discloses a data migration method for a hybrid storage architecture, see [link to relevant documentation]. Figure 1 As shown, the method includes: Step S11: Obtain multi-source heterogeneous data under the hybrid storage architecture, and extract features from the multi-source heterogeneous data to generate a data popularity score based on the extracted data features; the data popularity score is a quantitative indicator used to characterize the likelihood of data being accessed.

[0023] In this embodiment, to overcome the limitations of traditional storage systems that rely solely on a single feature to determine data accessibility, this step collects multi-source heterogeneous data from a hybrid storage architecture, including visual data (such as images, PDFs, and scanned documents), text data (such as Word documents), and structured data (such as JSON access logs), providing a data foundation for subsequent intelligent migration decisions. Furthermore, feature extraction and fusion are performed on this heterogeneous data, transforming the raw data into computer-quantifiable feature vectors to generate a popularity score for each piece of data. This popularity score is a quantitative indicator representing the likelihood of data access, reflecting the access value and importance of the data in the current business scenario. It is the core basis for determining whether data should be placed in a high-performance storage layer (such as SSD) or a low-cost storage layer (such as S3). Generally, a higher popularity score indicates that the data is "hotter" and should be placed in a high-performance storage layer for faster access; a lower score indicates that the data is "colder" and more suitable for migration to a low-cost storage layer.

[0024] Step S12: Using data popularity scores and storage system status information as inputs, generate a data migration strategy using a multi-agent reinforcement learning model trained based on a federated learning framework.

[0025] After calculating the data popularity score, the core task of this step is to address the problem that traditional static rules cannot adapt to complex business changes. Specifically, it does not rely on static rules but instead uses the data popularity score along with storage system status information (such as local SSD remaining capacity, S3 bandwidth usage, network latency, etc.) as input. It then uses reinforcement learning agents deployed on each storage node to dynamically generate data migration strategies. To achieve collaborative optimization among multiple nodes and protect data privacy, this invention introduces a federated learning framework: each node independently trains its local reinforcement learning model, sharing only encrypted model parameters or gradients. A central coordination server aggregates and generates a global model, which is then distributed to each node to update its local strategy. Through this mechanism, each storage node can achieve globally consistent optimization of the migration strategy while protecting its original local data from leaving the domain, effectively avoiding decision conflicts and resource waste between nodes.

[0026] Step S13: Use a generative artificial intelligence model to simulate future data access patterns to generate forward-looking migration instructions, and execute data migration operations between the first storage medium and the second storage medium according to the forward-looking migration instructions and data migration strategy; wherein the unit storage cost of the first storage medium is higher than that of the second storage medium, and the data access latency of the first storage medium is lower than that of the second storage medium.

[0027] In this embodiment, to balance data access performance and storage cost, the hybrid storage architecture is divided into a first storage medium and a second storage medium. The first storage medium is a high-speed storage layer with low access latency and high unit storage cost, suitable for storing frequently accessed hot data; the second storage medium is a low-speed storage layer with relatively high access latency and low unit storage cost, suitable for storing cold data with low access frequency.

[0028] As a specific implementation, the first storage medium can be a local solid-state drive (SSD) cache; the second storage medium can be a public cloud object storage layer (such as S3). However, the present invention is not limited to this; any two levels of storage media that can reflect the aforementioned performance and cost differences can respectively constitute the first and second storage media of the present invention. It is understood that the terms high-speed / low-speed are relative concepts, referring to a comparison of access latency and unit storage cost between the two, rather than absolute values. Those skilled in the art can select appropriate storage media to implement the functions of the first and second storage media according to the actual system design.

[0029] Traditional data migration techniques only trigger cache loading after data is accessed, resulting in high latency for sudden hot data access. In this embodiment, generative artificial intelligence technology is used to simulate possible future data access patterns and trigger forward-looking migration instructions based on the simulation results. Potential hot data is pre-loaded from remote object storage to local high-speed storage (such as SSD cache), thereby reducing sudden access latency and improving system response performance. This step does not rely on passive migration based on historical statistical patterns, but rather proactively predicts future business trends to achieve pre-deployment of data hotspots.

[0030] It's important to note that future data access patterns are essentially sets of multiple possible future access sequences generated by a diffusion model, conforming to historical patterns and the current state. It simulates the probability distribution and sequence of which data might be accessed under specific conditions. For example: a sequential pattern: the probability that a user accesses file A and then files B and C; a periodic pattern: the peak access to financial report-related files at the end of each month; a correlation pattern: after a certain business goes live, its related configuration, logs, and model files are accessed frequently simultaneously. Thus, it leverages the advantages of generative AI to provide the system with the ability to anticipate multiple future possibilities, thereby making more robust and proactive data preloading decisions to cope with sudden access peaks.

[0031] Finally, based on the generated forward-looking migration instructions and data migration strategies, the storage interface is actually invoked to complete the data movement between local storage and object storage, and local caching is managed. Specifically, data preloading is performed according to the forward-looking migration instructions, and data migration is performed according to the data migration strategy. This step covers the migration of cold data to low-cost object storage and the retention or preloading of hot data to local SSDs, which can automatically and efficiently complete the flow of cold and hot data between different storage media, achieving a dynamic balance between storage cost and access performance.

[0032] The beneficial effects of this application are as follows: First, by collecting multi-source heterogeneous data and extracting features, the generated data popularity score can accurately quantify data popularity, solving the single-dimensional evaluation defect of traditional methods that only rely on access frequency, and significantly improving the accuracy of hot data identification. Second, by adopting a decision-making mechanism that combines federated learning and multi-agent reinforcement learning, in a multi-node distributed storage scenario, through the federated learning framework, each storage node only shares encrypted model gradients rather than the original data, effectively solving the risk of data privacy leakage in cross-node collaborative training. At the same time, the multi-agent reinforcement learning architecture can dynamically adjust the migration strategy according to the real-time business load, overcoming the defect that traditional static threshold rules cannot adapt to sudden business traffic. Finally, it innovatively introduces a generative artificial intelligence model to simulate future potential access patterns, realizing forward-looking migration: cold data is migrated to low-cost storage media, significantly reducing storage costs while ensuring data accessibility; while predicted hot data is preloaded to high-performance storage media, forming a combination strategy of "low-cost storage + high-performance caching". In this way, the performance advantages of the first storage medium and the cost advantages of the second storage medium are effectively balanced, the sudden access latency is greatly reduced, and the global optimization of storage cost and access performance is achieved while ensuring data consistency.

[0033] like Figure 2 The diagram shown is an exemplary system architecture diagram for implementing data migration in a hybrid storage architecture, as provided in this embodiment. It includes a data acquisition module 01, a strategy decision module 02, a migration execution module 03, and a status verification module 04. The functions of each module are as follows: Data Acquisition Module 01: Receives and parses multimodal data, such as file paths, access logs, and metadata information. It supports heterogeneous data input such as images, text, and logs. It combines OCR (Optical Character Recognition) and NLP (Natural Language Processing) technologies to extract multidimensional features of the data. It intelligently calculates data popularity scores to provide a reference for subsequent data migration decisions. Strategy Decision Module 02: Generates migration strategies based on data characteristics and storage status, dynamically determines whether specific data needs to be migrated to cloud S3 storage or preloaded to local storage, supports multi-node collaborative decision-making, and resolves strategy conflicts; Migration Execution Module 03: Performs data migration or local preloading. This module calls storage interfaces (such as the S3 storage API) to migrate cold data and manages local preloading; Status Verification Module 04: Verifies migration results and provides status information. It ensures data consistency through hash verification and records key metrics such as migration success rate and storage usage.

[0034] Overall, this system consists of four main modules: data acquisition, strategy decision-making, migration execution, and status verification. It covers the entire data migration process. Data acquisition is responsible for analyzing data characteristics, collecting multimodal data, and extracting structured features. Strategy decision-making generates migration instructions and forms migration decisions by analyzing data characteristics and storage status. The migration execution module calls specific storage interfaces to complete data migration and local preloading operations. Finally, the status verification module verifies the migration results throughout the process and provides real-time feedback to ensure data consistency.

[0035] Based on the above embodiments, this application describes the technical process implemented by the data acquisition module 01. For example... Figure 3 As shown, specifically, this module is used to perform the following steps: Step 1: Obtain multi-source heterogeneous data under a hybrid storage architecture through the application programming interface or local file system; the multi-source heterogeneous data includes visual data, text data and structured data, and supports batch processing and real-time streaming input; Step 2: Extract features from the visual data to generate image semantic features; image semantic features include text content features extracted by calling optical character recognition algorithms to recognize characters and then performing keyword matching on the recognized characters using regular expressions, as well as visual structure features extracted by convolutional neural networks; Step 3: Vectorize the text data and input the vectorized data into a natural language processing model to generate text semantic features; Step 4: Perform metadata parsing and normalization on the structured data to generate metadata features; Step 5: Dynamically assign weights to image semantic features, text semantic features, and metadata features through an attention mechanism to perform feature weighted fusion and generate a data popularity score.

[0036] In this embodiment, the module reads data through an API interface or a local file system, supporting batch processing and real-time streaming input. Multi-source heterogeneous data includes, but is not limited to, visual data (such as images, screenshots, scanned documents, etc., common formats being PDF, JPEG, and PNG), text data (such as technical documents in Word format), and structured data (such as access log data recorded in JSON format).

[0037] In this embodiment, differentiated feature extraction algorithms are employed for multi-source heterogeneous data: For visual data, features are extracted to generate image semantic features. In one feasible implementation, an OCR algorithm is invoked to identify key characters in the text region of the image, and then regular expressions are used to match business keywords to finally extract text content features; in another feasible implementation, a convolutional neural network (CNN) is used to extract convolutional features from the image data to obtain visual structural features. For text data, the text is vectorized and input into an NLP algorithm model to extract text semantic features. In particular, a pre-trained NLP model is used to perform semantic analysis on the document content to identify high-frequency access patterns as semantic clues to determine whether the document belongs to hot data. For example, a document that repeatedly uses words such as "tender" and "quotation" may have high short-term access popularity. For structured data, the storage system API interface is invoked to obtain metadata, and basic file attributes, including creation time, modification frequency, file type, and other metadata information, are parsed. Numerical normalization is performed on these attributes to eliminate differences between data of different dimensions, serving as the basic parameters for data popularity scoring.

[0038] Furthermore, the extracted image semantic features, text semantic features, and metadata features are input into a large language model. The model uses an attention mechanism to dynamically allocate the weights of each modality feature to adapt to different business scenarios. In one feasible implementation, the data popularity score can be calculated using the following weighted formula: Data Popularity Score = α × Access Frequency + β × File Type Weight + γ × Content Keyword Matching Degree; where α, β, and γ are adjustable parameters that can be automatically adjusted through training with historical data; the access frequency is obtained from the metadata features, the file type weight is determined according to a preset mapping table based on text semantic features, and the content keyword matching degree is determined based on image semantic features.

[0039] Independently evaluating the popularity score of each file may overlook the access correlations between files (such as a project depending on multiple configuration files, or a user continuously accessing a series of images). Therefore, in a feasible implementation, the system can also introduce a graph neural network to propagate the popularity of related files. Specifically, a graph neural network popularity propagation module can be added. This module constructs a data association graph, where nodes represent data files and edges represent co-occurrence relationships or business dependencies between files (for example, establishing edges between files continuously accessed by the same user over a period of time). A graph convolutional network (GCN) is used to aggregate and update node features, so that when a file is marked as a hotspot, the predicted popularity of its neighboring nodes also increases accordingly. The propagated popularity values ​​are used to trigger batch preloading of related data, avoiding page misses in subsequent related accesses due to loading only a single file. Thus, this module can improve the preloading hit rate in sequential access scenarios and reduce multiple cross-layer migrations.

[0040] Finally, after completing the data popularity score, the data popularity score is combined with other attributes of the file (such as file information, file category, etc.) into a feature vector form, which is then passed to the strategy decision module.

[0041] As can be seen, the above steps achieve a leap from single-dimensional perception to multimodal panoramic perception. Compared to traditional technologies that only look at "who is accessed more often," this solution can also identify "who is more important." For example, a file that has not been frequently accessed recently but contains core business keywords can be accurately identified as hot data, thereby ensuring that critical data is not mistakenly migrated to low-speed storage media, providing a solid data foundation for subsequent accurate migration decisions.

[0042] Based on the above embodiments, this application describes the technical process implemented by the strategy decision module 02. For example... Figure 4 As shown, specifically, this module is used to perform the following steps: Step 1: Define each storage node in the hybrid storage architecture as an agent, and set up a state space, action space, and reward function for each agent; the state space includes data popularity score, local storage remaining capacity, access frequency, and network bandwidth usage; the action space includes data migration, data retention, and data preloading; the reward function calculates the reward value based on the changes in storage cost and access latency after migration. Step 2: Collect local running data of each agent and use the Q-learning algorithm to train their respective reinforcement learning models locally to generate local model parameters and model gradient information; Step 3: Obtain the model gradient information uploaded by each agent through the central coordination server, and use homomorphic encryption technology to encrypt and aggregate the model gradient information to generate global model parameters and distribute them to each agent; wherein, the encryption and aggregation process satisfies the differential privacy protection mechanism. Step 4: Update the local model parameters of each agent using the global model parameters so that each agent can generate a data migration strategy based on the updated reinforcement learning model.

[0043] First, the input feature vector and storage status are considered. This module's function is to input data, receive data popularity scoring information, and obtain various status indicators of the storage system in real time (such as local SSD remaining space, S3 bandwidth usage, etc.). The two types of data are then concatenated to form the strategy input data.

[0044] In this embodiment, a multi-agent collaborative model is adopted, abstracting each physical storage node in the hybrid storage architecture as an independent agent. For each agent, the following elements are set: State space: including data popularity score and storage status (such as remaining local storage capacity, access frequency, and network bandwidth usage); Action space: including three types of actions: data migration (moving data to object storage), data retention (keeping data locally), and data preloading (pulling data back to local storage in advance). This diverse action space design allows the strategy to simultaneously consider cost and performance. Reward function: Calculates reward values ​​based on changes in storage cost and access latency after executing migration actions. For example, migrating cold data to S3 reduces storage costs and gives a positive reward; keeping hot data locally improves access performance and also gives a positive reward; if the data migration strategy leads to performance degradation or cost increase, a negative reward is given.

[0045] Each agent employs the Q-learning algorithm for policy optimization. Specifically, the policy is updated using reinforcement learning algorithms such as Q-tables or neural network Q-Learning. After sufficient exploration, the agent can learn the transfer policy that maximizes cumulative rewards and generate data transfer instructions.

[0046] Because the hybrid storage architecture contains multiple storage nodes, to achieve a globally optimal strategy, agents do not directly share raw data (to protect privacy). Therefore, this invention introduces a federated learning framework to implement distributed node aggregation training. The specific process is as follows: First, each storage node independently updates the parameters of its reinforcement learning model or calculates gradients locally using its collected state-action-reward data. During training, the raw data remains locally and is not transmitted externally. Second, the central coordination server collects gradient information from each node. During this process, the system performs homomorphic encryption to ensure that the server can still perform mathematical calculations without being able to see the raw gradient plaintext. Finally, the central coordination server distributes the encrypted and aggregated global model parameters to each storage node, and the corresponding agent on each node uses these parameters to update its local reinforcement learning model. The above process can be executed periodically (e.g., once per hour), allowing the models on each node to gradually converge to consistency while protecting data privacy, thereby achieving a globally optimal data migration strategy.

[0047] It should be noted that, to further prevent the reverse engineering of original data through gradients, the encrypted aggregation process adheres to differential privacy protection mechanisms. Specifically, the system adds Gaussian-distributed random noise (e.g., ε=1.5) to the locally computed gradient data. This parameter setting significantly increases the difficulty for attackers to reconstruct a specific user's original data from the aggregated model while ensuring the model's training convergence accuracy, thus meeting enterprise-level storage compliance requirements. Homomorphic encryption is then used to encrypt and transmit the noise-added gradient data, ensuring that the original data remains within its domain during training.

[0048] Finally, based on the generated data migration strategy, specific data migration instructions (such as migrating DATA-14 to the cloud S3) are generated and passed to the execution module.

[0049] In one feasible implementation, to avoid conflicting operations on the same data object when multiple nodes generate migration decisions simultaneously, this invention designs a dedicated Multi-Node Negotiation Protocol (MCP) for consensus and conflict arbitration. A two-phase commit (2PC) mechanism is employed to ensure instruction uniqueness.

[0050] Following conflict arbitration, a data migration operation is performed between the first and second storage media. To adapt to dynamic changes in workload, this invention also designs a dynamic policy update mechanism: based on changes in the first and second storage media, the data migration strategy is dynamically adjusted to update the reward function parameters corresponding to the agent.

[0051] In this embodiment, the data migration strategy is dynamically adjusted according to changes in the storage system environment. For example, metrics such as remaining local storage space and remote object storage bandwidth usage are monitored in real time. When the remaining local storage space is below a preset threshold, the migration priority of cold data (such as PDF files) is increased to maximize efficiency in freeing up local storage space for higher-priority data. When the remote object storage bandwidth usage is above a preset threshold, the concurrency of batch migrations is reduced. The results of these priority adjustments are fed back into the reinforcement learning reward function, enabling the agent's strategy to adaptively adjust with environmental changes.

[0052] After the above processing, when each storage node receives a data migration request, it inputs the current data's popularity score and the system's storage status into its local reinforcement learning model. The model then outputs a specific migration decision. This decision is passed to the migration execution module, triggering subsequent data migration or preloading operations. It is evident that the introduction of the federated learning framework solves the privacy challenge in cross-node data collaboration. Through a distributed architecture of encrypted gradient updates and local model training, it ensures the privacy of data across nodes while achieving collaborative optimization of the global model. This design effectively reduces the security risks associated with cross-node data transmission and, through conflict negotiation mechanisms (such as the MCP protocol), reduces data synchronization conflicts between nodes, significantly improving the collaborative efficiency of the distributed system. Through the deep integration of multi-agent reinforcement learning and federated learning, this invention achieves distributed intelligent decision-making under a hybrid storage architecture, protecting data privacy while ensuring the consistency and adaptability of the global strategy, realizing efficient collaboration among multiple nodes, and significantly improving the intelligence level of data migration.

[0053] Based on the aforementioned embodiments, this application describes the technical process implemented by the migration execution module 03. For example... Figure 5 As shown, specifically, this module is used to perform the following steps: Step 1: Use historical access logs as training data and a diffusion model as a generative artificial intelligence model to learn the distribution of data access patterns. Step 2: Based on the current business status of the storage system, generate the target number of future data access patterns through a diffusion model.

[0054] In this embodiment, a diffusion model is first constructed and trained as a generative AI model. Historical access logs from the long-term operation of the storage system are collected, and these logs are used to simulate data access trends under potential business scenarios using new AI technologies such as Generative Adversarial Networks (GANs). After training, the diffusion model can randomly generate multiple possible future data access pattern scenarios based on the current business state of the storage system.

[0055] Step 3: Evaluate the data migration costs and business benefits under each data access mode to determine the prediction confidence of the diffusion model output; Step 4: When the prediction confidence is greater than the preset threshold, generate a forward migration instruction, use the Least Recently Used algorithm to manage the local solid-state drive cache to limit the maximum local cache capacity, and preload the predicted hot data from the public cloud object storage layer to the local solid-state drive cache according to the forward migration instruction. Step 5: According to the data migration strategy, call the remote object storage interface to migrate the predicted cold data to the public cloud object storage layer, and retain the original file copy of the cold data in the public cloud object storage layer.

[0056] After simulating future data access patterns, the system calculates the data migration cost versus business benefit ratio for each scenario. The system monitors the prediction confidence level output by the diffusion model in real time; this confidence level characterizes the proportion of data files predicted as hotspots across N scenarios. Only when the prediction confidence level exceeds a preset threshold is the system identified as a high-confidence potential hotspot and generates a forward-looking migration instruction for that file. This mechanism avoids bandwidth waste and cache pollution caused by blind preloading.

[0057] Forward-looking migration instructions are used for cache space management and preloading: The Least Recently Used (LRU) algorithm is employed to manage the cache, limiting the maximum local cache size. Based on forward-looking migration instructions, frequently accessed data is pre-loaded into the local SSD cache, reducing latency during sudden access spikes. Data migration strategies are used to execute data migrations: Migration to the public cloud object storage layer is achieved using AWS S3 interfaces or enterprise-developed interfaces, with migration logs recorded. Cold data is migrated via the S3 API, supporting basic object storage functions such as breakpoint resume and version control. Simultaneously, a copy of the migrated cold data is stored in the public cloud object storage layer, preserving the original file versions.

[0058] As can be seen, the cold and hot data tiered management mechanism enables precise allocation of storage resources. Migrating cold data to low-cost storage media (such as S3) significantly reduces storage costs while ensuring data accessibility; while the hot data pre-loading technology dynamically predicts high-probability access data and prioritizes its deployment on high-performance storage layers (such as SSDs), forming a "low-cost storage + high-performance caching" combined strategy. This two-tiered storage architecture controls infrastructure investment and improves overall system performance through the pre-deployment of hot data, achieving synergistic optimization of cost and performance. The proactive migration mechanism monitors system operating status and business load changes in real time, combining predictive models to plan data migration paths in advance. When a potential hot event is detected, the system can quickly initiate the pre-loading process, deploying critical data to the optimal location on the access path. This proactive response strategy overcomes the limitations of traditional passive migration, enabling the system to handle sudden access peaks with ease, minimizing latency while ensuring service quality, and significantly improving business continuity assurance capabilities.

[0059] Based on the above embodiments, this application describes the technical process implemented by the status verification module 04. For example... Figure 6 As shown, specifically, this module is used to perform the following steps: Step 1: Perform hash verification on the data before and after the migration; Step 2: If the source hash value before migration matches the target hash value after migration, the migration is marked as successful, and a consistency report is generated; the consistency report includes migration status, changes in storage costs, and verification results; Step 3: If the source hash value before migration does not match the target hash value after migration, record an error log and trigger a retransmission mechanism.

[0060] In this embodiment, after the migration execution module completes the migration of data from the local solid-state drive cache to the public cloud object storage layer, or the preloading from the public cloud object storage layer to the local solid-state drive cache, the state verification module is responsible for comparing the data before and after the migration to ensure that the data has not been damaged or lost during transmission. Simultaneously, this module records key indicators such as verification results, migration success rate, and changes in storage usage, and feeds them back to the policy decision module as sample data in the reinforcement learning agent's experience replay pool for subsequent incremental model training and policy adjustment. Through this closed-loop feedback mechanism, the system can continuously optimize itself, improving the accuracy and efficiency of migration decisions.

[0061] This module monitors the status output of the migration execution module in real time. Once the migration execution module completes a data migration or preloading task, it sends a completion message to the status verification module. To ensure data consistency and prevent tampering or corruption during transmission, the status verification module uses a hash checker and the SHA-256 algorithm to perform hash verification on the data before and after migration. It compares the source hash value before migration with the target hash value after migration. If they match, the data consistency verification is successful; otherwise, the verification fails, and a retransmission mechanism is triggered, sending a re-migration command to the migration execution module until consistency is achieved or the preset retry limit is reached.

[0062] Upon completion of verification, the state verification module generates a consistency report, which includes at least the following: migration status and changes in storage costs (e.g., a 10% decrease in overall storage costs after migration) and the verification result (pass / fail). Furthermore, the state verification module records migration logs in real time. This log data not only supports subsequent auditing and backtracking but can also serve as an offline training dataset, periodically used to retrain or fine-tune generative AI prediction models and reinforcement learning models, enabling long-term adaptive evolution of the system.

[0063] Through the above steps, this invention achieves closed-loop verification and feedback optimization after migration operations, ensuring the integrity and consistency of data in the hybrid storage architecture, while providing a data foundation for continuous improvement of the intelligent decision-making module, significantly enhancing the reliability and adaptability of the system.

[0064] In summary, the various technical modules of this invention are not isolated, but rather form an organic whole through a closed-loop architecture of intelligent prediction, dynamic migration, and collaborative optimization. Generative AI-driven prediction provides intelligent support for data migration decisions, the federated learning framework ensures the continuous evolution of the prediction model, the MCP protocol ensures the reliability of multi-node collaboration, and the hot and cold data tiering mechanism lays the physical foundation for the implementation of this invention. This multi-dimensional technical collaboration enables the system to possess self-awareness, self-decision-making, and self-optimization capabilities, constructing a new paradigm of elastic storage architecture for future business needs.

[0065] Accordingly, this application also discloses a data migration device with a hybrid storage architecture, see [link to relevant documentation]. Figure 7 As shown, the device includes: The data popularity score generation module 11 is used to acquire multi-source heterogeneous data under the hybrid storage architecture, and to extract features from the multi-source heterogeneous data to generate a data popularity score based on the extracted data features; the data popularity score is a quantitative indicator used to characterize the likelihood of data being accessed. The data migration strategy generation module 12 is used to take data popularity score and storage system status information as input and generate a data migration strategy using a multi-agent reinforcement learning model trained based on a federated learning framework. The data migration module 13 is used to simulate future data access patterns using a generative artificial intelligence model to generate forward-looking migration instructions, and to execute data migration operations between the first storage medium and the second storage medium according to the forward-looking migration instructions and the data migration strategy; wherein the unit storage cost of the first storage medium is higher than that of the second storage medium, and the data access latency of the first storage medium is lower than that of the second storage medium.

[0066] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0067] Therefore, the above-described scheme in this embodiment achieves several advantages. First, by collecting multi-source heterogeneous data and extracting features, the generated data popularity score can accurately quantify data popularity, solving the single-dimensional evaluation defect of traditional methods that rely solely on access frequency, and significantly improving the accuracy of hot data identification. Second, by adopting a decision-making mechanism that combines federated learning and multi-agent reinforcement learning, in a multi-node distributed storage scenario, the federated learning framework ensures that each storage node only shares encrypted model gradients rather than the original data, effectively solving the risk of data privacy leakage in cross-node collaborative training. Simultaneously, the multi-agent reinforcement learning architecture can dynamically adjust migration strategies according to real-time business load, overcoming the defect that traditional static threshold rules cannot adapt to sudden business traffic. Finally, it innovatively introduces a generative artificial intelligence model to simulate future potential access patterns, achieving forward-looking migration: cold data is migrated to low-cost storage media, significantly reducing storage costs while ensuring data accessibility; while predicted hot data is pre-loaded to high-performance storage media, forming a combined strategy of "low-cost storage + high-performance caching". In this way, the performance advantages of the first storage medium and the cost advantages of the second storage medium are effectively balanced, the sudden access latency is greatly reduced, and the global optimization of storage cost and access performance is achieved while ensuring data consistency.

[0068] In one feasible implementation, the data popularity rating generation module 11 is specifically used for: Multi-source heterogeneous data under a hybrid storage architecture can be obtained through application programming interfaces or local file systems; the multi-source heterogeneous data includes visual data, text data and structured data, and supports batch processing and real-time streaming input; Visual data is subjected to feature extraction to generate image semantic features; image semantic features include text content features extracted by calling optical character recognition algorithms to recognize characters and then performing keyword matching on the recognized characters using regular expressions, as well as visual structure features extracted by convolutional neural networks. The text data is vectorized, and the vectorized data is then input into a natural language processing model to generate text semantic features. Metadata parsing and normalization are performed on structured data to generate metadata features; The attention mechanism dynamically assigns weights to image semantic features, text semantic features, and metadata features to generate data popularity scores through weighted feature fusion.

[0069] In one feasible implementation, the data migration strategy generation module 12 is specifically used for: Each storage node in the hybrid storage architecture is defined as an agent, and a state space, action space, and reward function are set for each agent. The state space includes data popularity score, local storage remaining capacity, access frequency, and network bandwidth usage. The action space includes data migration, data retention, and data preloading. The reward function calculates the reward value based on the changes in storage cost and access latency after migration. Collect local operational data from each agent and use the Q-learning algorithm to train their respective reinforcement learning models locally to generate local model parameters and model gradient information; The model gradient information uploaded by each agent is obtained through a central coordination server, and homomorphic encryption technology is used to encrypt and aggregate the model gradient information to generate global model parameters and distribute them to each agent; the encryption and aggregation process satisfies the differential privacy protection mechanism. The local model parameters of each agent are updated using global model parameters, so that each agent can generate a data migration strategy based on the updated reinforcement learning model.

[0070] In one feasible implementation, the data migration module 13 includes: The conflict arbitration module is used to arbitrate conflicts when multiple storage nodes simultaneously perform data migration operations on the same data object, through a preset multi-node negotiation protocol; the preset multi-node negotiation protocol adopts a two-phase commit mechanism. The strategy adjustment module is used to perform data migration operations between the first storage medium and the second storage medium after conflict arbitration, and dynamically adjust the data migration strategy according to the changes in the first storage medium and the second storage medium to update the reward function parameters corresponding to the agent.

[0071] The model training module uses historical access logs as training data and a diffusion model as a generative artificial intelligence model to learn the distribution of data access patterns. Based on the current business status of the storage system, it generates a target number of future data access patterns through the diffusion model. The forward migration instruction generation module is used to evaluate the data migration cost and business benefits under various data access modes to determine the prediction confidence of the diffusion model output. When the prediction confidence is greater than a preset threshold, a forward migration instruction is generated to preload the predicted hot data to the first storage medium.

[0072] In one feasible implementation, the data migration module 13 is specifically used for: The local SSD cache is managed using the Least Recently Used algorithm to limit the maximum local cache capacity, and predicted hot data is preloaded from the public cloud object storage layer to the local SSD cache based on forward-looking migration instructions. According to the data migration strategy, the remote object storage interface is called to migrate the predicted cold data to the public cloud object storage layer, and the original file copy of the cold data is retained in the public cloud object storage layer.

[0073] In one feasible implementation, the data migration apparatus for a hybrid storage architecture further includes: The consistency verification module is used to perform hash verification on the data before and after migration. If the source hash value before migration matches the target hash value after migration, the migration is marked as successful and a consistency report is generated. The consistency report includes the migration status, changes in storage costs, and verification results. If the source hash value before migration does not match the target hash value after migration, an error log is recorded and a retransmission mechanism is triggered.

[0074] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0075] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data migration method of the hybrid storage architecture disclosed in any of the foregoing embodiments.

[0076] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0077] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it can include an operating system 221, computer programs 222, and data 223, etc. The data 223 can include various types of data. The storage method can be temporary storage or permanent storage.

[0078] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data migration method of the hybrid storage architecture executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0079] Furthermore, this application also discloses a computer-readable storage medium, which includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, optical disks, or any other form of storage medium known in the art. The computer program, when executed by a processor, implements the aforementioned data migration method for a hybrid storage architecture. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0080] Furthermore, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the above-described methods for data migration of the hybrid storage architecture.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0082] The steps of the data migration method or algorithm for the hybrid storage architecture described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0083] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0084] The above provides a detailed description of the data migration method, apparatus, device, and medium for a hybrid storage architecture provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data migration method for a hybrid storage architecture, characterized in that, include: Acquire multi-source heterogeneous data under a hybrid storage architecture, and extract features from the multi-source heterogeneous data to generate a data popularity score based on the extracted data features; the data popularity score is a quantitative indicator used to characterize the likelihood of data being accessed. The data popularity score and storage system status information are used as inputs to generate a data migration strategy using a multi-agent reinforcement learning model trained based on a federated learning framework. Generative artificial intelligence models are used to simulate future data access patterns to generate forward-looking migration instructions. Based on the forward-looking migration instructions and the data migration strategy, data migration operations between a first storage medium and a second storage medium are executed. The unit storage cost of the first storage medium is higher than that of the second storage medium, and the data access latency of the first storage medium is lower than that of the second storage medium.

2. The data migration method for a hybrid storage architecture according to claim 1, characterized in that, The process of acquiring multi-source heterogeneous data under a hybrid storage architecture and extracting features from the multi-source heterogeneous data to generate a data popularity score based on the extracted data features includes: Multi-source heterogeneous data under a hybrid storage architecture can be obtained through application programming interfaces or local file systems; wherein, the multi-source heterogeneous data includes visual data, text data and structured data, and supports batch processing and real-time streaming input; Feature extraction is performed on the visual data to generate image semantic features; the image semantic features include text content features extracted by calling an optical character recognition algorithm to recognize characters and then performing keyword matching on the recognized characters using regular expressions, as well as visual structure features extracted by a convolutional neural network. The text data is vectorized, and the vectorized data is input into a natural language processing model to generate text semantic features. The structured data is parsed and normalized to generate metadata features; The image semantic features, text semantic features, and metadata features are dynamically assigned weights through an attention mechanism to perform feature weighted fusion and generate a data popularity score.

3. The data migration method for a hybrid storage architecture according to claim 1, characterized in that, The step of using the data popularity score and storage system status information as input, and generating a data migration strategy using a multi-agent reinforcement learning model trained based on a federated learning framework, includes: Each storage node in the hybrid storage architecture is defined as an agent, and a state space, action space, and reward function are set for each agent. The state space includes the data popularity score, local storage remaining capacity, access frequency, and network bandwidth usage. The action space includes data migration, data retention, and data preloading. The reward function calculates the reward value based on the changes in storage cost and access latency after migration. Collect local operating data of each agent and use the Q-learning algorithm to train their respective reinforcement learning models locally to generate local model parameters and model gradient information; The model gradient information uploaded by each intelligent agent is obtained through a central coordination server, and the model gradient information is encrypted and aggregated using homomorphic encryption technology to generate global model parameters and distribute them to each intelligent agent; wherein the encryption and aggregation process satisfies the differential privacy protection mechanism. The local model parameters corresponding to each agent are updated using the global model parameters, so that each agent can generate a data migration strategy based on the updated reinforcement learning model.

4. The data migration method for a hybrid storage architecture according to claim 3, characterized in that, In the process of performing the data migration operation between the first storage medium and the second storage medium according to the forward migration instruction and the data migration strategy, the method further includes: When multiple storage nodes simultaneously perform data migration operations on the same data object, conflict arbitration is carried out through a preset multi-node negotiation protocol; the preset multi-node negotiation protocol adopts a two-phase commit mechanism. After conflict arbitration, a data migration operation is performed between the first storage medium and the second storage medium, and the data migration strategy is dynamically adjusted according to the changes in the first storage medium and the second storage medium to update the reward function parameters corresponding to the agent.

5. The data migration method for a hybrid storage architecture according to claim 1, characterized in that, The method of using generative artificial intelligence models to simulate future data access patterns to generate forward-looking migration instructions includes: Historical access logs are used as training data, and a diffusion model is used as a generative artificial intelligence model to learn the distribution of data access patterns. Based on the current business status of the storage system, the diffusion model is used to generate a target number of future data access patterns. The data migration costs and business benefits under each of the aforementioned data access modes are evaluated to determine the prediction confidence level of the diffusion model output. When the prediction confidence level is greater than a preset threshold, a forward migration instruction is generated to preload the predicted hotspot data into the first storage medium.

6. The data migration method for a hybrid storage architecture according to claim 1, characterized in that, The first storage medium includes a local solid-state drive cache, and the second storage medium includes a public cloud object storage layer; The step of performing a data migration operation between the first storage medium and the second storage medium according to the forward migration instruction and the data migration strategy includes: The local solid-state drive cache is managed using the Least Recently Used algorithm to limit the maximum local cache capacity, and predicted hot data is preloaded from the public cloud object storage layer to the local solid-state drive cache according to the forward migration instructions. According to the data migration strategy, the remote object storage interface is invoked to migrate the predicted cold data to the public cloud object storage layer, and the original file copy of the cold data is retained in the public cloud object storage layer.

7. The data migration method for a hybrid storage architecture according to any one of claims 1 to 6, characterized in that, After performing the data migration operation between the first storage medium and the second storage medium according to the forward migration instruction and the data migration strategy, the method further includes: Perform hash verification on the data before and after the migration; If the source hash value before migration matches the target hash value after migration, the migration is marked as successful, and a consistency report is generated; the consistency report includes migration status, changes in storage costs, and verification results; If the source hash value before migration does not match the target hash value after migration, an error log is recorded and a retransmission mechanism is triggered.

8. A data migration device with a hybrid storage architecture, characterized in that, include: The data popularity score generation module is used to acquire multi-source heterogeneous data under a hybrid storage architecture, and to extract features from the multi-source heterogeneous data to generate a data popularity score based on the extracted data features; the data popularity score is a quantitative indicator used to characterize the likelihood of data being accessed. The data migration strategy generation module is used to take the data popularity score and storage system status information as input and generate a data migration strategy using a multi-agent reinforcement learning model trained based on a federated learning framework. The data migration module is used to simulate future data access patterns using a generative artificial intelligence model to generate forward migration instructions, and to execute data migration operations between a first storage medium and a second storage medium according to the forward migration instructions and the data migration strategy; wherein the unit storage cost of the first storage medium is higher than that of the second storage medium, and the data access latency of the first storage medium is lower than that of the second storage medium.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the data migration method of the hybrid storage architecture as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used for storing computer programs; wherein the computer programs, when executed by a processor, implement the data migration method of the hybrid storage architecture as described in any one of claims 1 to 7.

Images