Data tiered storage method and device based on access characterization, equipment and medium

Abstract

The application relates to the technical field of data processing, can be applied to business scenes such as financial technology and medical health, and discloses a data hierarchical storage method and device based on access representation, equipment and a medium, which comprises the following steps: collecting access behavior features of original data, identifying the heat state, writing data into a matched first target storage equipment according to a preset equipment performance corresponding relationship; after data storage, continuously monitoring subsequent access features, dynamically judging whether the heat state changes; if the heat state changes and meets specific migration conditions, a second target storage equipment is re-allocated and a data migration operation is completed, so that automatic hierarchical storage management of cold and hot data is realized. Through the acquisition of data access representation and the dynamic identification of the heat state, hierarchical migration of different heat data between different performance storage equipment is realized, the resource use efficiency and the access performance of the storage system are improved, and the storage cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data hierarchical storage method, apparatus, device, and storage medium based on access representation. Background Technology

[0002] In today's data-driven enterprise environment, with the increasing complexity of business systems and the continuous expansion of data volume, enterprises are placing higher demands on efficient and intelligent storage systems. Traditional Network Attached Storage (NAS) systems typically use fixed-configuration storage media, such as uniformly using HDDs or SSDs, employing the same storage strategy regardless of data access frequency. This "one-size-fits-all" approach ignores the differences in data access behavior, resulting in a large amount of cold data occupying high-performance storage resources for extended periods. This not only increases unnecessary costs but also weakens the access efficiency of frequently accessed data, thus hindering the overall performance of the storage system.

[0003] In the fintech business, institutions typically need to process high-concurrency transaction data, risk control model logs, user interaction behavior, and other "hot" data in real time, while also retaining a large amount of infrequently accessed "cold" data such as historical vouchers and audit records. Existing storage systems often cannot effectively identify and differentiate the access characteristics of this data, resulting in core transaction data and audit archive data being written together to high-performance storage media, causing performance congestion, resource waste, and affecting real-time risk control and customer response efficiency.

[0004] In the healthcare sector, electronic medical records, imaging data, and monitoring data constitute massive and diverse medical datasets. These include both "hot" data—diagnostic reports and vital sign curves that require frequent access in the short term—and "cold" data—large amounts of long-term archived case records and follow-up documents. Traditional storage systems lack dynamic scheduling and hierarchical management capabilities when dealing with this mixed hot and cold data storage scenario. This leads to performance bottlenecks such as slow loading and access conflicts in critical applications like image retrieval and medical record analysis, severely impacting clinical work efficiency and data service quality.

[0005] Furthermore, while some existing technologies incorporate rule engines to configure tiered data storage, they largely rely on manually setting access frequency thresholds and static policy matching. This fails to adapt to the fluctuations in data access characteristics caused by rapid changes in business models and lacks the ability to analyze and predict data access behavior in real time. Such static strategies struggle to respond promptly to changes in data popularity, often resulting in cold data residing on high-performance storage media for extended periods or hot data remaining on low-speed storage media, thus limiting the intelligence and adaptability of the storage system.

[0006] Therefore, existing technologies still have significant shortcomings in areas such as data access characteristic identification, dynamic hierarchical management of hot and cold data, and optimized allocation of storage resources. More intelligent and refined solutions are urgently needed to improve the overall storage efficiency and service performance of the system. Summary of the Invention

[0007] The main objective of this invention is to provide a data hierarchical storage method, apparatus, device, and storage medium based on access characteristics, aiming to solve the technical problem that existing technologies cannot dynamically identify the data popularity status based on data access behavior and automatically adjust the storage medium allocation, resulting in mixed storage of hot and cold data and low resource utilization efficiency.

[0008] To achieve the above objectives, the present invention provides a data hierarchical storage method based on access representation, comprising:

[0009] Obtain the initial data access representation of the raw data;

[0010] Based on the initial data access characterization, the data access characteristics of the original data are identified through the first data analysis and processing to generate the initial data popularity status of the original data.

[0011] Based on the initial data popularity status and the preset storage device performance correspondence, a first target storage device is allocated to the original data, and the original data is stored in the first target storage device;

[0012] Monitor subsequent data access characteristics of the data already stored in the first target storage device;

[0013] Based on the subsequent data access characterization, the second data analysis and processing determines whether the data popularity status of the stored data has changed compared to the initial data popularity status;

[0014] When the data popularity status of the stored data changes and the preset data movement conditions are met, a second target storage device is allocated to the stored data, and the stored data is moved from the first target storage device to the second target storage device.

[0015] Furthermore, to achieve the above objectives, the present invention provides a data hierarchical storage device based on access representation, comprising:

[0016] The access representation acquisition module is used to obtain the initial data access representation of the raw data;

[0017] The access characteristic identification module is used to identify the data access characteristics of the original data based on the initial data access characterization and to generate the initial data popularity status of the original data through the first data analysis and processing.

[0018] The initial write decision module is used to allocate a first target storage device to the original data according to the initial data heat status and the preset storage device performance correspondence, and store the original data into the first target storage device;

[0019] The access behavior monitoring module is used to monitor subsequent data access characteristics of the data already stored in the first target storage device;

[0020] The heat status determination module is used to determine, based on the subsequent data access characteristics, whether the heat status of the stored data has changed compared to the initial data heat status through a second data analysis process.

[0021] The storage migration execution module is used to allocate a second target storage device to the stored data and move the stored data from the first target storage device to the second target storage device when the data popularity status of the stored data changes and the preset data movement conditions are met.

[0022] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an access-based hierarchical storage program stored in the memory and executable on the processor, wherein the access-based hierarchical storage program, when executed by the processor, implements the steps of the access-based hierarchical storage method as described above.

[0023] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data hierarchical storage program based on access representation, wherein the data hierarchical storage program based on access representation, when executed by a processor, implements the steps of the data hierarchical storage method based on access representation as described above.

[0024] Beneficial Effects: This invention relates to the field of data processing technology and can be applied to business scenarios such as fintech and healthcare. It discloses a data hierarchical storage method, apparatus, device, and medium based on access representation, comprising: acquiring initial data access representations of raw data; identifying the initial data popularity state of the raw data based on the access representations; matching the corresponding first target storage device according to the initial data popularity state and storing the raw data therein; continuously monitoring subsequent data access representations of the data already stored in the first target storage device and determining whether the popularity state has changed; when the popularity state changes and a preset data movement condition is met, reallocating a second target storage device and migrating the stored data from the first target storage device to the second target storage device. This invention, by acquiring data access representations and dynamically identifying popularity states, achieves hierarchical migration of data with different popularity levels between storage devices with different performance levels, improving the resource utilization efficiency and access performance of the storage system, and reducing storage costs. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0026] Figure 1 This is a schematic diagram of an application environment for a data hierarchical storage method based on access representation in one embodiment of the present invention;

[0027] Figure 2 This is a flowchart illustrating an embodiment of the data hierarchical storage method based on access representation of the present invention;

[0028] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data hierarchical storage device based on access representation of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] The data hierarchical storage method based on access representation provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can obtain the initial data access characterization of the original data from the user terminal, identify the initial data popularity status of the original data based on this access characterization, match the corresponding first target storage device according to the initial data popularity status, and store the original data there. It continuously monitors the subsequent data access characterization of the data already stored in the first target storage device and determines whether the popularity status has changed. When the popularity status changes and preset data movement conditions are met, a second target storage device is reallocated, and the stored data is migrated from the first target storage device to the second target storage device. This invention achieves hierarchical migration of data with different popularity levels between storage devices with different performance levels by obtaining data access characterization and dynamically identifying popularity status, thereby improving the resource utilization efficiency and access performance of the storage system and reducing storage costs. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The invention will be described in detail below through specific embodiments.

[0033] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the data hierarchical storage method based on access representation provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the data hierarchical storage method based on access representation proposed in this invention includes the following steps:

[0035] S10, Obtain the initial data access representation of the raw data;

[0036] In this embodiment, obtaining the initial data access characterization of the raw data mainly involves collecting and processing various types of information recording the access behavior of the raw data to form a dataset that characterizes the data access characteristics. The raw data may include structured data, unstructured data, semi-structured data, or log-like data, and its sources include databases, distributed file systems, data lake platforms, etc. The "access characterization" in the initial data access characterization is a composite feature set designed to express the access patterns of the raw data across dimensions such as time, frequency, method, and interval.

[0037] To implement this process, a time period must first be set during which access log information of the raw data is continuously collected. This access log records the specific actions of reading and writing data, including access types such as read, write, modify, and delete. The access log should also record the timestamp of the access operation, the identifier of the accessing entity, the source IP of the access, and the success or failure status of the access, for subsequent calculations.

[0038] Next, based on the collected log data, the total number of times the original data was accessed within a unit time window is calculated to form the access frequency characteristic. The unit time window can be set according to specific business needs, such as 5 minutes, 1 hour, or 24 hours. The access frequency characteristic is used to reflect the activity level of the original data within a certain time period and its density of computing resource usage.

[0039] In addition to access frequency, it is also necessary to evaluate the interval between consecutive accesses of the raw data. This is done by calculating the time difference between two adjacent accesses and taking the average of these differences to generate time interval characteristics. Time interval characteristics can reveal the concentration and regularity of data access; a smaller average usually indicates that the data accesses are high-frequency and continuous.

[0040] In addition, it is necessary to statistically analyze the distribution ratio of different types of access operations, such as read operations accounting for 70%, write operations accounting for 20%, and delete operations accounting for 10%. The distribution ratio of operation types is an important dimension in access characterization to reflect differences in data usage patterns, and it has important decision-making value, especially in data migration and hot / cold data classification.

[0041] Finally, the access frequency characteristics, time interval characteristics, and operation type distribution ratios mentioned above are integrated to construct a complete initial data access representation. This representation can take the form of a vector, structured record, or multidimensional tensor, and can be used for subsequent data analysis and popularity determination.

[0042] In implementation, access log collection can be accomplished by a data access monitoring agent deployed in the storage system or business middleware. This agent can be injected into the business call chain via a hook mechanism or achieve non-intrusive collection through the operating system's file system event capture interface. Data access records are written to a log database or message queue, and then parsed in real time by a log processing engine (such as Flink or Spark).

[0043] The access frequency calculation module can be implemented based on a sliding window or a fixed window, using a counter to accumulate visits and calculating the total number of visits after the window ends. The time interval calculation can be performed by differencing and averaging a list of access timestamps sorted by time. The distribution statistics of access operation types can be aggregated using MapReduce to aggregate the operation type field.

[0044] The data feature integration module can uniformly call the calculation results of three types of features—access frequency, time interval, and operation type—in the middle layer, and construct them into JSON format or vectorized representation and store them in the metadata system.

[0045] Example Explanation: In the healthcare business, electronic medical records, examination image data, and patient behavior logs generated in hospital information systems are often stored in a unified system. By collecting access behavior data in different departments and time periods, and constructing access frequency and interval characteristics, it is possible to accurately distinguish between real-time diagnosis and treatment-related "hot" data and archived "cold" data, supporting subsequent automatic archiving or speed-up services.

[0046] In the fintech business, there are significant differences in the access characteristics of customer transaction records, risk control model logs, and compliance audit data. Obtaining the initial access characteristics of raw data can help identify hot data nodes in high-frequency transactions, optimize their distributed storage structure, improve data reading efficiency, and perform cold storage processing on low-frequency access audit data to reduce resource consumption.

[0047] This embodiment, by acquiring initial data access characteristics of the raw data, can comprehensively capture the access behavior patterns of the raw data in the actual operating environment. Without relying on manual labeling or preset rules, it provides quantifiable input for subsequent popularity status identification and storage strategy optimization. Especially in distributed data environments, this approach avoids strong coupling with specific business logic, providing high-quality feature input for subsequent introduction of AI models to process access behavior.

[0048] S20, based on the initial data access characterization, the data access characteristics of the original data are identified through the first data analysis and processing to generate the initial data heat status of the original data;

[0049] In this embodiment, analysis using initial data access characterization identifies the access characteristics of the original data and generates an initial data popularity status, which is a core action in the data processing chain that connects the preceding and following steps. The initial data access characterization aggregates access frequency characteristics, time interval characteristics, and access operation type distribution characteristics. The analysis and processing here requires extracting the access patterns of the original data based on multi-dimensional information to make it categorizable, and then inferring the popularity status through a model.

[0050] First, data cleaning is required for the initial data access representation. During data access log collection, issues such as missing access counts, abnormal access timestamps, and non-standard operation type field values ​​may occur. Therefore, missing value imputation, outlier filtering, and standardization are necessary. Missing value imputation can be achieved using methods such as moving average or median replacement; abnormal access operation types can be removed using a whitelist mechanism; standardization typically employs z-score normalization to ensure that all feature dimensions have the same scale, which is beneficial for subsequent model convergence.

[0051] After data preprocessing, the results are fed into a pre-trained model for classification inference. This model is a multi-class classification model, trained with labeled historical visit representations and popularity status mapping samples, and can employ algorithms such as random forests, gradient boosting trees, and neural networks. The model output is a multi-class probability distribution, representing the confidence level of the original data belonging to different visit popularity levels. For example, hot data represents active and frequently visited data, cold data represents data with very few visits or long periods of inactivity, and warm data falls between the two, exhibiting periodic or sudden visit characteristics.

[0052] Finally, based on the principle of maximum confidence, the raw data is categorized into one of three types, and its popularity status is marked. Simultaneously, to support subsequent data scheduling operations, a unique identifier value needs to be generated for the popularity status result and written into the corresponding metadata description, forming a callable field for the data management system.

[0053] In actual deployment, the data cleaning module is typically implemented using Spark or Pandas, and its processing flow can be defined and executed as a standard operation chain within the data access pipeline. Abnormal operation type judgment can be automatically executed based on a manually defined set of legal operations, combined with a regular expression matching and filtering mechanism. The standardization module can set global feature statistics to maintain consistency across different batches of data processing.

[0054] Training of the classification model can be completed during system initialization or periodic retraining. Training samples are derived from cross-records of access behavior and storage distribution status in the historical storage system. During deployment, the model is compiled into ONNX format and loaded by TensorRT or other inference engines in the inference service. The model inference service can interface with the main system via gRPC, supporting batch access to representation inputs.

[0055] The classification probability vector output by the model is classified with the highest probability using a decision function. If the classification confidence is too low, it can be marked as uncertain and re-identified after subsequent multi-period data collection. The classification results will be injected into the global metadata index via a RESTful interface or metadata management engine and bound to the identifier of the original data.

[0056] Example Explanation: In healthcare data scenarios, different patients exhibit varying access patterns to their medical records, imaging data, and vital sign monitoring logs. By analyzing their initial data access characteristics, we can identify "hot" data currently being treated and "cold" data from archived cases. Appropriately classifying their access levels helps improve online access performance and reduce the burden on backend storage.

[0057] In the financial sector, customer behavior logs, account change records, and transaction flow data exhibit strong access heterogeneity. The system uses the aforementioned processing methods to identify initial activity levels, accurately distinguishing between high-frequency data during peak trading periods and static risk control parameter tables. This provides high-performance support for real-time response requests while migrating low-activity data to more cost-effective storage resources.

[0058] This embodiment transforms access representations into categorizable popularity states, enabling not only modeling of usage behavior for raw data but also providing quantitative criteria for subsequent hierarchical storage scheduling. This method improves the accuracy and robustness of popularity identification, avoids the limitations of relying on static rules that cannot adapt to dynamic access changes, ensures that data under different access modes is reasonably classified, and establishes the foundation for subsequent resource matching strategies.

[0059] S30, based on the initial data heat status and the preset storage device performance correspondence, allocate a first target storage device to the original data and store the original data in the first target storage device;

[0060] In this embodiment, after identifying the data access frequency, matching the original data with a suitable storage device requires a decision based on the coupling relationship between data characteristics and device capabilities. The data access frequency reflects comprehensive characteristics such as access frequency, time intervals, and operational behaviors, while the storage device's performance indicators include read / write speed, access latency, I / O concurrency, unit storage cost, and energy consumption. This matching process aims to optimize resource utilization by matching hot data with high-speed devices and allocating cold data to low-cost devices.

[0061] In practice, the first step is to establish a mapping between storage device performance. This mapping can be obtained through a preset configuration file or a resource awareness mechanism, describing a matching matrix between popularity levels and device performance attributes. For example, high popularity corresponds to NVMe SSDs, medium popularity corresponds to SATA SSDs, and low popularity corresponds to HDDs or distributed archiving devices. This matrix can be established by combining the device's benchmark performance test results and cost evaluation parameters, and then cached as a rule set in the system.

[0062] Once the raw data is assigned a popularity status, the system filters all candidate devices that meet the requirements from the storage device resource pool based on this status. The device resource pool is a dynamically managed collection of devices, containing metadata such as the real-time available space, current load rate, health status, and category of each device.

[0063] Next, a selection process needs to be performed from the candidate set. This selection logic is typically based on a weighted scoring mechanism, comprehensively considering the device's unit storage cost, read / write performance score, and current load balancing coefficient to calculate a comprehensive priority score. The scoring function can incorporate device aging factors and cold / hot migration cost factors as fine-tuning parameters. The device with the highest score becomes the primary target storage device for the current raw data.

[0064] After allocation, a secure data transmission link is established to ensure that data is not tampered with, leaked, or lost during network transmission. Link security is typically achieved through TLS, IPSec, or channel protection mechanisms based on SDP architecture. Data transmission employs block-level segmentation and parallel writing to match the throughput capacity of the target device.

[0065] After the data is successfully stored, the system synchronously updates the global metadata index, mapping and registering the original data identifier with the identifier of the first target storage device. This operation ensures data addressability and supports subsequent data access scheduling, status monitoring, and migration operations.

[0066] A set of device scheduling service nodes can be deployed to periodically synchronize the operating status and performance parameters of each storage device, maintaining a unified device capability database. This database provides a query interface, supporting the retrieval of device candidate sets by popularity category. Data popularity status is transmitted from the upstream identification module to the device scheduling module, triggering device filtering and allocation.

[0067] The optimization algorithm can dynamically adjust weight parameters based on custom strategy configurations, such as increasing the weight of read / write speeds in high-concurrency access scenarios and increasing the impact factor of unit capacity cost in cost-sensitive businesses. The optimization function can be deployed in the load balancing layer and calculated in real time by the system's built-in scoring model.

[0068] During data transmission, a write task command is sent to the target device management process via a message middleware. The write task is executed asynchronously and concurrently. A hash digest is recorded during transmission, and data integrity is verified on the storage device. After a successful write signal is received, the metadata management engine is triggered to record the device mapping relationship and complete the status annotation in the data visibility module.

[0069] Policy constraints can also be added during the target device selection process, such as specifying that multiple raw data under the same data category or business tag be written to the same physical location to reduce cross-device scheduling costs.

[0070] Example: In financial business scenarios, users experience high-frequency transaction logs, risk model iteration parameters, and strategy test data access pressure. High-speed write-type SSD devices can be matched according to their access frequency. Meanwhile, long-term archived contract scans and historical transaction data can be allocated to low-power cold storage devices, thereby ensuring a balance between response speed and cost.

[0071] In healthcare scenarios, real-time collected vital signs data and online image analysis cache need to be deployed to low-latency storage nodes to meet the requirements of second-level response. Meanwhile, data that requires long-term access, such as patient historical image archives and surgical records, are written to a large-capacity storage array after heat identification, which effectively alleviates the pressure on the main storage and reduces operating costs.

[0072] This embodiment precisely matches data popularity with device capabilities, ensuring that the storage location of raw data is highly consistent with its access pattern. This effectively mitigates the impact of high-frequency data on low-speed devices and prevents cold data from consuming high-performance resources, significantly improving the rationality of storage resource allocation and operational efficiency. Furthermore, an automatic device selection mechanism reduces manual intervention, enabling the implementation of an intelligent tiered storage strategy.

[0073] S40, monitor the subsequent data access characteristics of the data already stored in the first target storage device;

[0074] In this embodiment, continuous monitoring of subsequent access behavior to already stored data is a prerequisite for dynamically adjusting the data storage location. The essence of this process is to construct a behavior analysis system that changes over time to determine whether data access patterns have changed significantly, thereby providing a basis for data migration decisions.

[0075] First, a dynamic monitoring time window needs to be set for each data object. This time window is not fixed but dynamically adjusted based on the current popularity of the data. For example, for hot data, a shorter monitoring period is set to ensure high responsiveness and capture fine-grained behavioral changes; while for cold data, the monitoring period can be set to a longer period to reduce unnecessary system load. The adjustment mechanism of the time window can be implemented by referring to the sliding window strategy or the exponential decay function.

[0076] Within each dynamic monitoring time window, the system needs to accurately record data access events, primarily including read and write operations. The event log structure includes fields such as access timestamp, access type, and access initiator identifier, generating real-time access operation records. This record data structure can be stored in a lightweight memory cache, maintaining performance and stability through periodic disk writes.

[0077] Next, key access statistics are extracted from these access operation records. The number of read and write operation events reflects the usage frequency during the window, constituting a dynamic access frequency indicator; the average time difference between two adjacent accesses reflects the concentration and continuity of data usage, forming a dynamic time interval indicator. These statistical indicators provide the basic input for subsequent popularity judgment and trend prediction.

[0078] To capture the evolving direction of access trends, it is necessary to compare the current dynamic access frequency index with historical access frequency characteristics. The difference can be calculated using methods such as differencing algorithms, moving average change rates, or Z-score methods to generate an access trend change factor. This factor reflects the degree of deviation of data access frequency from historical performance and can provide a decision-making reference for the next stage of data popularity identification.

[0079] Finally, by integrating dynamic access frequency, dynamic time interval, and trend change factors, a multi-dimensional subsequent access representation vector is formed. This representation can not only quantify the current access behavior of the data, but also construct an access trend model, serving as a key input for subsequent time series analysis and popularity prediction.

[0080] An access listener deployment based on a proxy layer can be used to transparently capture and record all access operations after data is written to the device. The access listener can be integrated into the storage system's I / O path or deployed on the access layer middleware node, providing high-performance event filtering and data acquisition capabilities.

[0081] The dynamic configuration of the monitoring time window is achieved through metadata tagging. The popularity tag of each data point is associated with its monitoring period parameters, and the scheduling frequency of monitoring tasks is adjusted synchronously during updates. Monitoring scheduling is managed by an event-driven task scheduler, and the number of concurrent monitoring tasks can be controlled through a system load feedback mechanism.

[0082] Access event logs are cached using an in-memory database or a circular buffer, and a threshold is set to trigger the persistence process, avoiding frequent disk writes. Access frequency and time interval metrics are updated in real time by a sliding statistics module, and trend factors are calculated using a method that compares exponential moving averages with a periodic baseline.

[0083] It can also transmit subsequent access representation vectors to a centralized analysis node in real time, enabling cross-device data status aggregation and providing a unified input source for global storage optimization. For important data, it can also combine access subject permission information or system tags from the access source to achieve cross-dimensional access behavior analysis.

[0084] Example Explanation: In the fintech business, the access frequency of customer behavior logs, risk control parameter caches, and transaction matching data fluctuates dramatically with market activity. By setting up short-term dynamic monitoring windows to capture these access fluctuations in real time, it is possible to effectively identify active and inactive periods and achieve synchronized adjustments to data storage strategies.

[0085] In the healthcare field, real-time monitoring of vital signs data, image analysis results, and historical case visit behavior is crucial for identifying active and follow-up phases of patients. Monitoring cycle parameters can be set for different types of health data, and the granularity of data access trend assessment can be dynamically adjusted to rationally allocate medical data storage resources and improve the response efficiency and data availability of remote diagnosis and treatment systems.

[0086] This embodiment, through joint monitoring of dynamic time windows and multi-dimensional access metrics, enables the system to accurately capture changes in data usage at different lifecycle stages, effectively avoiding the sluggish response of traditional static classification models to changes in data status. Simultaneously, by incorporating the fluctuation trends of access behavior into the analysis model, the system enhances its predictive ability for future access changes, providing a more reliable data foundation for subsequent popularity status identification and migration strategies.

[0087] S50, based on the subsequent data access characterization, determine through the second data analysis and processing whether the data popularity status of the stored data has changed compared to the initial data popularity status;

[0088] In this embodiment, to identify the dynamic changes in data access patterns during the storage lifecycle, a second analysis and processing mechanism is introduced to quantitatively determine the trends of the data access behaviors already generated, thereby identifying whether the data popularity status has changed. The core lies in mapping subsequent access representation vectors to the probability space of popularity status changes, and using this to determine whether the fluctuation exceeds the tolerance threshold of the original popularity.

[0089] The first step in the analysis is to standardize the subsequent data access representations so that data from different time windows and access frequencies can be compared on the same scale. This standardization typically employs Z-score normalization, min-max compression, or baseline difference calculation methods to eliminate scaling effects or periodic errors in access behavior.

[0090] The processed, standardized access representations are input into a time series analysis model, which is trained based on historical access sequences and corresponding heat status change annotations. The input sequence structure consists of access frequencies and time intervals within multiple dynamic monitoring time windows, used to capture the continuity, abrupt changes, and trend patterns of access behavior. The annotation structure represents the heat change trend between each time period, such as from hot to warm, warm to cold, or stable.

[0091] Time series models can be constructed using Long Short-Term Memory (LSTM) networks, Variational Autoencoders (VAEs), or ARIMA (Autoregressive Moving Average) models to predict the fluctuation range of access frequency and time intervals in future time periods. The prediction results are used to construct the future state distribution of access behavior and can be compared with the current popularity threshold benchmark for deviation.

[0092] Deviation is quantified by calculating the distance between the predicted value and the threshold corresponding to the current popularity status, forming frequency fluctuation deviation and interval fluctuation deviation respectively. These two are then weighted to obtain a comprehensive deviation index. When this value exceeds a set threshold, it can be determined that the data popularity status has changed significantly.

[0093] This change determination is not only reflected in numerical limits exceeding limits, but also in generating data popularity status change instructions and binding them with metadata, forming logical trigger points for subsequent migration, routing adjustments and other operations, thus achieving intelligent adaptation of the storage structure.

[0094] A pre-trained LSTM time series model can be used to receive standardized access representations as input, and a lightweight inference engine deployed on cloud storage nodes can output future trend intervals in real time. The model automatically learns the fluctuation patterns under different popularity states based on a large number of access records collected during the training phase, and generalizes modeling according to different device loads and data types.

[0095] The deviation threshold can be set by selecting the optimal parameter through K-fold cross-validation, or it can be dynamically adjusted in the actual operating environment based on empirical statistics. The weighting factor can be set according to parameters such as the current device read / write pressure and the migration cost between target hot categories to adapt to migration constraints between different levels of storage.

[0096] Unsupervised clustering can also be used to automatically identify the boundaries of state changes based on the aggregation changes of access behavior, which is suitable for scenarios with insufficient data labels or insufficient historical data. In addition, to improve processing efficiency, the standardization and deviation calculation modules can be deployed in parallel on edge nodes, with asynchronous threads performing calculations and judgments in memory.

[0097] Example Explanation: In the fintech business, data such as risk scoring caches or payment behavior logs exhibit strong time-varying and periodic characteristics. By predicting fluctuations in access frequency, peak and inactive periods can be identified in advance, enabling dynamic data transfer between high-performance SSDs and cost-effective HDDs, thus ensuring an optimal balance between performance and cost in the fund clearing process.

[0098] In the healthcare sector, patient medical data is accessed frequently during hospitalization, but gradually becomes archived long-term after discharge. By analyzing the trends in subsequent access patterns, it's possible to determine which cases will enter the cold data zone, allowing for the timely release of high-performance storage space while ensuring the availability and consistency of historical access, thus optimizing the allocation of healthcare information system resources.

[0099] This embodiment effectively avoids the judgment errors based on static snapshots by using a time-series-based trend judgment and threshold comparison mechanism, enabling dynamic control over the evolution path of access behavior. By predicting future access states and establishing a benchmark comparison with initial popularity, it not only improves the accuracy of popularity identification but also achieves precise control over the timing of data migration, thereby enhancing the resource adaptation efficiency and response agility of the entire tiered storage system.

[0100] S60, when the data popularity status of the stored data changes and the preset data movement conditions are met, a second target storage device is allocated to the stored data, and the stored data is moved from the first target storage device to the second target storage device.

[0101] In this embodiment, when data access behavior changes significantly within a certain period, and this change has been identified as a shift in data usage intensity by the preceding analysis model, a cross-storage media data migration process should be initiated to prevent mismatched storage resources from continuing to be occupied. The goal of this part is to accurately match resources with data usage needs based on real-time dynamic adjustments of data usage intensity, thereby reducing storage costs or improving performance.

[0102] The system first obtains the generated data popularity status transition command and reads the metadata description information associated with the target data, especially the current storage location mapping. By analyzing this metadata structure, it can determine which physical storage device the data is currently located on, and further verify whether migration has been triggered.

[0103] Preset data movement conditions are used to avoid wasting system resources through frequent migration operations. These conditions can include two aspects: first, the current load status of the storage device, such as whether I / O pressure, CPU utilization, and the number of concurrent tasks exceed the threshold; and second, whether the data residency period reaches the maximum period set by the system. In particular, if the long-term residency characteristics of cold data in a cold and hot data tiered structure are not met, the storage level should be adjusted.

[0104] After determining that the above conditions are met, the system needs to match suitable storage devices according to the new popularity status. This is done by checking the performance parameters, load status, and cost configuration of each storage device in the resource pool, and then filtering the storage subset that corresponds to the target popularity category. Further, based on indicators such as read / write latency, throughput, and cost, a comprehensive priority is calculated, and the optimal device is selected as the second target storage device.

[0105] The data migration process requires the establishment of a highly reliable and secure data transmission channel to ensure that the data content is not tampered with or lost during the migration. After the transmission is completed, a data cleanup process must be triggered in the source device, and the global metadata index must be updated to write the new location mapping information and synchronize it to the metadata service to ensure the correct routing of subsequent access requests.

[0106] To avoid historical popularity status misleading subsequent access strategies, the current popularity status should be updated to the metadata management module simultaneously with the updated mapping information, serving as a reference for route adjustments and subsequent popularity judgments.

[0107] Metadata indexes can be maintained using a doubly linked list structure, allowing access to the popularity status, historical access records, and current storage path of each data object in O(1) time. The migration triggering module is deployed in the data scheduling layer, taking the output of the subsequent access analysis module as input, and scheduling migration tasks according to priority through a queue.

[0108] If the target storage devices are from different manufacturers or use different protocol platforms, a protocol adaptation layer can be introduced to unify the data transmission format, such as transcoding and encapsulation between S3 and POSIX, ensuring cross-platform consistency. Data migration can employ snapshot compression combined with transmission to improve throughput and stability during data transfer.

[0109] In a high-concurrency environment, multiple data migration tasks can be executed in parallel, based on file granularity or block granularity. The concurrency is controlled by an asynchronous thread pool, and a failure retransmission mechanism is set up to ensure reliability.

[0110] You can also set migration window priority strategies, such as prioritizing the migration of data that is about to reach its residency threshold or data that is becoming less popular, to optimize the release speed of high-performance device resources. At the same time, in conjunction with the access prediction module, you can delay the migration of some temporarily cold data that may cool down temporarily, avoiding repeated migrations.

[0111] Example Description: In the healthcare field, a large number of high-resolution medical images generated during patient examinations are initially in high-frequency use. Doctors and AI-assisted diagnostic systems frequently access these image files to support rapid diagnosis and multidisciplinary consultations. Based on access frequency and time interval analysis, the system identifies these as "hot data" and allocates them to high-performance storage devices. However, after the diagnostic cycle ends, the access frequency of these images drops significantly. The system detects this change in access pattern and, based on the current storage resource load, determines that migration conditions are met. It then migrates the image files from high-speed RAID disks to an object storage system, simultaneously updating the access path mapping table so that subsequent access commands are automatically routed to the new device, ensuring uninterrupted access consistency.

[0112] In the fintech business, intelligent risk control systems frequently read user transaction behavior logs during risk analysis cycles to perform real-time modeling. This data is initially identified as "hot data" and stored on low-latency SSDs to support high-concurrency reads. As risk assessments are completed, the data access frequency rapidly decreases. The system detects this change in access frequency and the high load on the SSDs, thus meeting the migration criteria. The platform then automatically compresses and migrates this batch of log data to hybrid cloud cold storage, cleans up historical copies on the source device, and frees up high-performance resources for the next round of risk model loading, optimizing resource recycling and processing efficiency.

[0113] This embodiment effectively prevents high-performance resources from being occupied by low-frequency data by migrating data with significantly changing access frequency from the current storage environment and re-matching it to a more suitable target device, thereby optimizing overall system performance and unit storage cost. Through precise judgment and controlled triggering, a dynamic balance between performance and cost is achieved, while reducing data access latency and improving system response efficiency and user experience.

[0114] This invention relates to the field of data processing technology and can be applied to business scenarios such as fintech and healthcare. It discloses a data hierarchical storage method, apparatus, device, and medium based on access representation, comprising: acquiring an initial data access representation of raw data; identifying the initial data popularity state of the raw data based on the access representation; matching a corresponding first target storage device according to the initial data popularity state and storing the raw data therein; continuously monitoring subsequent data access representations of the data already stored in the first target storage device and determining whether the popularity state has changed; when the popularity state changes and a preset data movement condition is met, reallocating a second target storage device and migrating the stored data from the first target storage device to the second target storage device. This invention, by acquiring data access representations and dynamically identifying popularity states, enables hierarchical migration of data with different popularity levels between storage devices with different performance levels, improving the resource utilization efficiency and access performance of the storage system and reducing storage costs.

[0115] In one embodiment, step S10 includes:

[0116] S101, collect data access logs of raw data within a preset time period, the data access logs including the access operation types of the raw data;

[0117] S102, Based on the data access log, determine the total number of accesses to the original data within a unit time window, and generate access frequency characteristics;

[0118] S103, Based on the data access log, determine the average time difference between two adjacent access operations of the original data, and generate time interval features;

[0119] S104, integrate the access frequency characteristics, the time interval characteristics, and the distribution ratio of the access operation types to generate the initial data access characterization of the original data.

[0120] In this embodiment, the process of collecting data access logs of raw data within a preset time period aims to build a foundation for behavioral-based data feature input. Access logs are structured event data generated in real time by the storage system or access control system, recording every read, write, modify, move, and copy operation on a data object. The preset time period can be flexibly set according to business needs, such as 1 hour, 6 hours, or 1 day; its selection will affect the representativeness of the logs and the stability of access patterns. The collection of access operation types should ensure coverage of various common access behaviors, such as labeling read operations as READ, write operations as WRITE, and modification operations as UPDATE. The system can categorize these operations using identifiers as input variables for subsequent statistics and analysis.

[0121] After obtaining the access logs, access counts need to be performed on each raw data object within a time window to calculate the total number of accesses within that time window, thereby generating an access frequency feature. The unit time window is a crucial parameter for segmenting time-series behavior, and its granularity setting must balance accuracy and computational resource overhead. For example, the time period can be subdivided into windows of 5 minutes, 15 minutes, or 30 minutes, and the number of access events within each window can be counted to form a time-series access frequency sequence. Then, the average or maximum value can be calculated as the frequency feature. A higher access frequency generally indicates that the data is in an active state.

[0122] Next, based on the same data access log, the time interval characteristics are further calculated, namely the average time difference between two adjacent access operations. By extracting the timestamp field, calculating the time series of access intervals, and then calculating their average, the rhythmic indicators of data access can be obtained. Data that is frequent but has regular intervals tends to have higher activity levels; while data with multiple accesses but extremely uneven interval distribution may be due to batch or asynchronous calls, requiring further identification and judgment based on the operation type.

[0123] Building upon the aforementioned frequency and time interval characteristics, further introducing the distribution ratio of access operation types reveals the usage patterns of raw data in scenarios such as read-heavy, write-heavy, high-frequency updates, or read-only archiving. Integrating these three elements forms a vectorized initial access representation, which serves as model input for the heat identification stage. Integration methods can include feature concatenation, weighted combination, principal component compression, or discrete encoding, standardizing access behavior into a unified structure to facilitate downstream analysis and storage strategy formulation.

[0124] This embodiment, through a combined modeling approach based on access frequency, access interval, and operation type distribution, comprehensively depicts the dynamic access behavior of raw data during business operations. This allows subsequent data popularity assessment to go beyond simple access volume statistics, incorporating multi-dimensional behavioral attributes to more accurately identify the boundary between active and inactive data. It can automatically adapt to differences in access characteristics under different business models, thereby laying a more stable and generalizable data foundation for subsequent data popularity grading and storage media allocation, achieving efficient matching of access behavior and storage resources.

[0125] In one embodiment, step S20 above includes:

[0126] S201, perform data preprocessing on the initial data access representation to generate a preprocessed initial data access representation;

[0127] S202, input the preprocessed initial data access characterization into the classification model trained based on historical data access characterization and corresponding data popularity status labels;

[0128] S203, perform pattern matching on the access characteristics of the original data using the classification model to generate the classification probability distribution of the original data;

[0129] S204, Based on the classification probability distribution, the original data is divided into one of the categories of hot data, warm data, or cold data to generate the initial data heat status;

[0130] S205, the initial data heat status is associated with the metadata description file of the original data in the form of a unique identifier.

[0131] In this embodiment, the main purpose of data preprocessing for the initial data access representation is to ensure that the feature data input into the classification model has structural integrity, consistency, and numerical comparability. Missing value imputation is used to handle gaps in fields such as access frequency or time interval caused by acquisition delays or system jitter, and methods such as moving average, global mean, and nearest neighbor imputation can be used. Abnormal access operation type filtering is used to exclude access records that may be introduced by system testing, erroneous calls, or malicious operations. Filtering rules may include operation type whitelists, frequency upper limit restrictions, or operation timing logic verification. Data normalization is used to unify the numerical scale of various access features, so that the model can avoid bias caused by differences in feature value dimensions. Methods such as min-max normalization, Z-score normalization, or logarithmic mapping can be used. After data preprocessing, the access behavior of the original data will be transformed into a standardized feature set with consistent structure and controlled noise.

[0132] The purpose of inputting the preprocessed initial data access representations into the classification model is to identify access patterns using existing experience. This classification model is generated through training using historical data access representations and their corresponding data popularity status labels. The model structure can be a support vector machine, decision tree, random forest, logistic regression, neural network, or an ensemble of these. The training objective is to establish a mapping relationship between access features and popularity categories. The model training process includes feature selection, cross-validation, and hyperparameter tuning to ensure generalization ability under different business contexts.

[0133] After receiving new access features, the classification model performs pattern matching and outputs the classification probability distribution for each popularity category. The classification probability distribution reflects the probability distribution of the original data under different popularity states; for example, hot data has a probability of 0.8, moderate data 0.15, and cold data 0.05, indicating that the data is highly active. After the classification probability distribution is generated, the system classifies the original data into specific popularity state categories based on the maximum probability principle or a multi-threshold strategy. Common categories include hot data, moderate data, and cold data, and can be further expanded or subdivided according to business scenarios.

[0134] Finally, the identified initial data popularity status is structurally represented as a unique identifier, and a one-to-one correspondence is established with the original data's metadata description file. The metadata description file typically stores all management information about the data in a key-value pair structure, including data name, generation time, data source path, and permission information. By embedding the popularity identifier into the metadata file, subsequent storage strategy selection, data migration management, and popularity change monitoring can be automated.

[0135] This embodiment improves the stability and accuracy of access features through data preprocessing mechanisms, and, in conjunction with a classification model trained on historical access behavior, achieves efficient identification of raw data access characteristics. It can dynamically determine data popularity based on multi-dimensional behavioral patterns and achieve refined data classification results. Furthermore, it structurally binds popularity status to metadata information, establishing a bridge between data access behavior and storage strategy execution, significantly improving the response efficiency and accuracy of subsequent popularity-aware migration.

[0136] In one embodiment, step S30 above includes:

[0137] S301, based on the initial data heat status, query the preset storage device resource pool information to match the corresponding storage device performance requirements, and filter out a set of candidate storage devices that meet the storage device performance requirements from the storage device resource pool information;

[0138] S302, determine the comprehensive priority score based on the read / write performance index and unit storage cost of each storage device in the candidate storage device set;

[0139] S303, Select the storage device with the highest comprehensive priority score as the first target storage device;

[0140] S304, Establish a secure data transmission channel with the first target storage device, and transmit the original data to the storage space of the first target storage device through the secure data transmission channel;

[0141] S305, when the original data is stored in the first target storage device, update the global metadata index library and record the storage mapping relationship between the original data and the first target storage device.

[0142] In this embodiment, the core of querying the preset storage device resource pool information based on the initial data popularity status lies in matching data popularity with storage device capabilities. The initial data popularity status is the result obtained from the previous stage through access characteristics identification, and can be labeled as hot data, warm data, or cold data, possessing enumerable characteristics. The storage device resource pool information is a structured data set containing multiple storage devices' performance indicators, storage types, current load status, storage paths, available capacity, energy consumption levels, and other meta-attributes. The query process establishes a mapping relationship between the popularity status and the device performance configuration table defined in the resource pool. For example, hot data corresponds to high IOPS and low latency devices (such as NVMe SSDs), while cold data corresponds to large-capacity, low-cost devices (such as nearline SATA HDDs). The filtering operation forms a candidate storage device set that meets performance requirements through a multi-level link of popularity status → performance requirements → device capabilities → filter set.

[0143] Based on the read / write performance metrics and unit storage cost of each device in the candidate storage device set, a comprehensive priority score is calculated to further optimize resource allocation strategies while ensuring heat adaptation. Read / write performance metrics mainly include parameters such as random read / write IOPS, sequential throughput, write latency, and concurrent processing capabilities. Unit storage cost includes equipment procurement cost, operating energy consumption, and maintenance frequency. After normalizing the performance metrics and cost factors, a weighted scoring function is constructed to achieve the comprehensive priority score. The scoring function can set weight parameters according to application preferences to achieve performance-oriented or cost-oriented strategy switching, such as prioritizing performance improvement in high-frequency trading or compressing budget consumption in archive backup scenarios.

[0144] The device with the highest overall priority score is designated as the primary target storage device for the initial disk write of the raw data. This device not only meets the service quality requirements needed for the current data volume but also offers the best cost-effectiveness and scheduling weight within the resource pool. This decision-making process can employ static rule mapping or combine a dynamic scoring model to adjust priority weights in real time, achieving both timeliness and flexibility in resource scheduling.

[0145] Establishing a secure data transmission channel with the primary target storage device is fundamental to ensuring the reliability of the data writing process. Secure data transmission channels are typically built upon mechanisms such as TLS encryption, VPN tunneling, SDN isolation, and segmented verification to prevent unauthorized access, data leakage, and tampering attacks during data transmission from the processing node to the target device. This channel can also incorporate a load-aware mechanism to dynamically allocate bandwidth resources and avoid transmission congestion.

[0146] After the raw data is successfully written to the first target storage device, the system automatically updates the global metadata index, marking the mapping relationship between the data and the device. The global metadata index is a unified management structure across devices and popularity levels, recording information such as the unique identifier, storage location, popularity category, storage timestamp, and access permissions for each piece of data. The update operation ensures that the system can quickly locate the target data when performing popularity changes, data migrations, and access scheduling, guaranteeing the consistency and integrity of data lifecycle management.

[0147] This embodiment achieves fine-grained allocation of raw data during the initial write phase by constructing a heat-driven storage device selection mechanism and combining device performance indicators with a cost trade-off model. It dynamically adapts device capabilities based on access characteristics and establishes a unified metadata index structure, creating a tight coupling between data resources and hardware resources. This approach improves access speed for high-frequency data, reduces storage costs for cold data, and enhances system security by introducing encrypted transmission mechanisms during the data write process, providing a solid foundation for subsequent data migration, scheduling, and heat management. By comprehensively considering performance, cost, and security factors, it significantly improves the overall efficiency and sustainability of the storage system.

[0148] In one embodiment, step S40 above includes:

[0149] S401, for the data already stored in the first target storage device, set a dynamic monitoring time window that is dynamically adjusted according to the initial data heat status;

[0150] S402, within the dynamic monitoring time window, capture every read operation event and write operation event of the stored data, and generate a real-time access operation record;

[0151] S403, count the number of read operation events and write operation events of the stored data within the dynamic monitoring time window, and generate the dynamic access frequency index of the current monitoring window;

[0152] S404, determine the average time difference between two adjacent access operations of the stored data within the dynamic monitoring time window, and generate a dynamic time interval index for the current monitoring window.

[0153] S405, Analyze the difference between the dynamic access frequency index and the historical access frequency characteristics to generate an access trend change factor;

[0154] S406, integrate the dynamic access frequency index, the dynamic time interval index, and the access trend change factor to generate subsequent data access characterization.

[0155] In this embodiment, setting a dynamic monitoring time window for the data already stored in the first target storage device is a key preliminary operation in data popularity lifecycle management. The dynamism of this time window is reflected in setting different monitoring frequencies and durations based on the initial data popularity status. Hot data corresponds to a shorter window period to respond promptly to changes in popularity, while cold data is configured with a longer period to reduce resource consumption. The time window can be set as a fixed-length window or a sliding window, generated through a rule table based on popularity classification or a dynamic policy mapping mechanism. After the data is written to disk, it is bound to the metadata control policy of the corresponding data object, achieving personalized access monitoring and scheduling.

[0156] Within the dynamic monitoring time window, the system captures all read and write operation events related to the stored data. The capture process is based on real-time monitoring of the I / O request queue, typically implemented through file system hooks at the operating system level, log replay interfaces of distributed storage engines, or access recording modules at the block storage layer, supporting low-latency event acquisition under high-frequency data streams. Each access event record includes elements such as access type, timestamp, request size, request source, and session ID, forming a structured real-time access operation record. This record serves as the core input source for subsequent access analysis, directly reflecting the actual behavioral state of data usage.

[0157] Based on collected access records, the system counts the number of read and write operations within the current time window to calculate a dynamic access frequency metric. This metric is measured in terms of the number of accesses per unit of time, supports statistical analysis by access type, and can also perform read / write weighted processing to reflect the interaction density and usage activity of data within the current period. Unlike the initial access frequency characteristics, the dynamic metric has phased and real-time characteristics, used to support trend judgment and the identification of changes in popularity.

[0158] In addition to access frequency, it is also necessary to evaluate the dynamic changes in operation time intervals. Specifically, this involves calculating the average time difference between two consecutive access operations to generate a dynamic time interval index for the current monitoring window. This index reflects the consistency or suddenness of user access behavior and is of key reference value when analyzing data cooling down from a hot state or activating from a cold state. If the time interval becomes significantly shorter, it may indicate that the data has been reactivated; if it becomes significantly longer, it may indicate that the data has entered a cooling-off phase.

[0159] Based on a comparison of current and historical access frequency metrics, the system generates an access trend change factor. This factor is obtained by calculating the difference between the current frequency and the initial frequency, the slope of change, or a weighted volatility coefficient, quantifying the direction and intensity of trend changes in data access behavior. For example, a continuous decline indicates a cooling trend, while an increase in volatility indicates a short-term hot trend. This factor provides intermediate characteristics to support subsequent judgments on changes in popularity.

[0160] Finally, the dynamic access frequency index, dynamic time interval index, and access trend change factor are integrated to form the subsequent data access representation. This representation structure maintains dimensionality consistent with the initial access representation, facilitating subsequent model invocation for state comparison analysis, and supporting fine-grained time series trend modeling or classifier input. This representation not only serves for popularity identification but also provides data support for potential future data migration, cache adjustment, and pre-fetch strategies.

[0161] This embodiment constructs an access behavior acquisition framework through a dynamic time window mechanism, enabling real-time tracking of data popularity evolution over time and avoiding the problem of static monitoring mechanisms failing to identify popularity changes in a timely manner. The combination of access frequency and operation interval metrics improves the sensitivity and accuracy of behavior recognition, while the introduction of access trend change factors enhances the system's ability to perceive behavioral trends. Throughout the entire processing flow, the monitoring operation covers the complete path of behavior acquisition, feature generation, trend evaluation, and structured output, providing a real-time, interpretable, and context-sensitive input basis for subsequent data popularity judgment, migration decisions, and resource allocation. This improves the responsiveness to the dynamics of storage resource allocation and enhances the system's efficiency and accuracy in identifying and processing data lifecycle evolution.

[0162] In one embodiment, step S50 above includes:

[0163] S501, The subsequent data access representation is standardized to generate a standardized subsequent data access representation.

[0164] S502, the standardized subsequent data access representation is input into a time series analysis model trained and generated based on the historical data access representation time series and the corresponding data popularity status change labels.

[0165] S503, using the time series analysis model, generate the predicted access frequency fluctuation range and the predicted time interval fluctuation range of the stored data within a future preset time window;

[0166] S504, compare the predicted access frequency fluctuation range with the benchmark access frequency threshold corresponding to the initial data popularity state to generate frequency fluctuation deviation.

[0167] S505, compare the predicted time interval fluctuation range with the benchmark time interval threshold corresponding to the initial data heat state to generate the interval fluctuation deviation.

[0168] S506, Based on the weighted sum of the frequency fluctuation deviation and the interval fluctuation deviation, determine whether the data popularity status of the stored data has changed;

[0169] S507, when the weighted sum exceeds a preset deviation threshold, it is determined that the data popularity status has changed, and a data popularity status change instruction is generated, and the data popularity status change instruction is associated with the metadata description file of the stored data.

[0170] In this embodiment, to determine whether the data popularity status of the stored data has changed compared to the initial data popularity status, the subsequent data access representation first needs to be standardized. This standardization process aims to eliminate differences in the numerical dimensions of different feature dimensions, enabling the subsequent analysis model to more accurately identify feature change trends. Standardization methods may include z-score standardization, min-max normalization, or quantile scaling. By using a unified standard scale, indicators such as access frequency and time intervals can be compared and integrated within a unified range, reducing the impact of feature skewness on the analysis results.

[0171] The standardized subsequent data access representation is input into the time series analysis model, based on learning and predicting the changing trends of continuous access behavior. The historical data access representation time series used in the model training process consists of a combination of access frequency and time interval indicators from multiple continuous dynamic monitoring time windows, ensuring the analysis model's ability to identify periodic, abrupt, or trend patterns. The corresponding heat status transition labels indicate whether the data changes from hot data to cold data or from cold data to hot data between adjacent time windows, serving as output labels in the supervised learning process to guide the model in establishing a mapping relationship between changes in data behavior and heat evolution.

[0172] Time series analysis models can utilize deep learning architectures such as Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Variational Autoencoder (VAE), or traditional prediction models such as ARIMA and exponential smoothing. The model outputs the predicted range of visit frequency fluctuations and the predicted range of time interval fluctuations within a preset future time window. In other words, it provides a range of data usage behavior changes over a forward-looking period, serving as an important basis for judging whether trends have shifted.

[0173] Next, the predicted range of access frequency fluctuations is compared with the baseline access frequency threshold corresponding to the initial data popularity state to calculate the frequency fluctuation deviation. This deviation is typically measured by the maximum deviation rate, mean deviation rate, or the degree of overlap of fluctuation intervals, reflecting the extent to which current usage behavior deviates from the original expected popularity. Similarly, the predicted range of time interval fluctuations is compared with the time interval baseline under the initial popularity state to generate the interval fluctuation deviation, assessing whether the temporal continuity of data use and the access interval have changed significantly.

[0174] These two deviation indicators are weighted and combined to form a weighted sum used to determine whether the data popularity status has changed. The weighting coefficients can be adjusted based on preset settings for the influence of frequency and interval on business scenarios to adapt to the characteristics of different data types and their sensitivity to timeliness and access density. By setting a preset deviation threshold as the decision boundary, when the weighted sum exceeds the threshold, it can be determined that the data popularity status has changed.

[0175] Finally, the system generates a data popularity status transition command to drive subsequent data migration, strategy adjustment, and other processes. This command is written into the original data's metadata description file in the form of structured metadata, ensuring that the system can perform actions such as storage reallocation based on this command during subsequent calls, supporting intelligent management of the storage lifecycle.

[0176] This embodiment employs a time-series modeling-based prediction mechanism, enabling the determination of popularity status not only to rely on current access behavior but also to incorporate historical trends and future evolution directions, achieving a forward-looking perception of popularity shifts. Standardized processing unifies feature dimensions, improving model generalization ability and judgment accuracy. Weighted fusion of frequency deviation and time interval deviation ensures a comprehensive evaluation of multi-dimensional access behavior, reducing the possibility of misjudgments from single indicators. The final generated popularity shift instructions provide a basis for subsequent data storage strategies, effectively enhancing the system's responsiveness to changes in the data lifecycle and the dynamic adaptability of resource scheduling.

[0177] In one embodiment, step S60 above includes:

[0178] S601, when the data popularity status of the stored data changes, obtain the data popularity status change instruction and the associated metadata description file of the stored data;

[0179] S602, based on the storage mapping relationship of the first target storage device recorded in the metadata description file, verify whether the preset data movement conditions are met;

[0180] S603, when the preset data movement condition is met, according to the target data heat state indicated in the data heat state change instruction, a second target storage device matching the target data heat state is selected from the storage device resource pool information;

[0181] S604, send a data migration instruction to the first target storage device, and transfer the stored data in the first target storage device to the storage space of the second target storage device through a secure data transmission channel;

[0182] S605, after the data migration is completed, delete the data in the first target storage device, update the storage mapping relationship in the global metadata index library, and record the mapping information between the data and the second target storage device;

[0183] S606, Adjust the routing strategy for subsequent data access requests based on the read / write performance metrics of the second target storage device.

[0184] In this embodiment, when a data popularity status changes, the system first obtains the generated data popularity status change instruction and the metadata description file associated with the data. The metadata description file typically contains the data's unique identifier in the system, current storage location, historical access characteristics, and policy identifiers, etc., to comprehensively reflect the data status and its mapping relationship with the infrastructure, ensuring accurate positioning and consistent information during subsequent migration.

[0185] The system verifies preset data migration conditions based on the mapping relationship between the first target storage device and the stored data recorded in the metadata description file. This set of conditions includes, but is not limited to: whether the current load rate of the first target storage device exceeds a preset load threshold, and whether the actual residence time of the stored data exceeds a preset period threshold. The current load rate can be calculated from I / O operation density, CPU utilization, or bandwidth utilization, and the residence period can be calculated from the data write time and the current system time. If any condition is met, the data is deemed to meet the migration requirements.

[0186] After the data movement conditions are met, the system selects devices whose performance indicators match the target data popularity status specified in the transformation instruction, based on the storage device resource pool information, as the second target storage device. The resource pool information typically includes the response latency, throughput, reliability level, cost-effectiveness ratio, etc. of each device, and the matching strategy may adopt methods such as weighted comprehensive scoring, priority ranking, or popularity interval mapping.

[0187] After selecting the target device, the system sends a data migration command to the first target storage device, and the data is transferred through a secure data transmission channel. This channel can be established based on SSL / TLS protocol, IPsec tunnel, or dedicated storage network channel (such as iSCSI, Fibre Channel) to ensure that the data is protected from leakage or tampering during the migration process. During the migration, data can be split and synchronized at the block, file, or object level to improve concurrent migration efficiency and fault tolerance.

[0188] After the data has been completely migrated to the second target storage device, the system performs the original data deletion operation after confirming the consistency of the replicas, releasing the storage space on the first target storage device. Following this, the system updates the global metadata index, replacing the mapping relationship between the original data and the new device, and synchronously records the change time, migration status, and target device identifier to ensure the traceability and accuracy of subsequent system scheduling and queries.

[0189] To ensure efficient access after data migration, the system also needs to adjust the routing strategy for subsequent access requests based on the read / write performance metrics of the second target storage device. This adjustment includes IO scheduling logic within the load balancer or file system, optimizing access paths based on the new storage location so that requests can directly reach the new target device, avoiding redundant jumps or pointer invalidation, thereby maintaining system performance stability and response consistency.

[0190] Example Description: In the fintech business, a large bank's data management platform needs to implement hierarchical management and intelligent scheduling of its historical transaction data, customer behavior logs, and risk control model input data to reduce high-frequency storage resource consumption and improve the retrieval efficiency of important data. The platform deploys a distributed intelligent storage system with hot / cold tier management capabilities. The system first obtains the initial data access characteristics of the aforementioned raw data (such as user transaction records and loan application logs). To this end, the platform collects access logs for this data over the past month, extracting the access operation type (such as query, update), access frequency per unit time, and average interval between adjacent accesses for each data entry. This is then integrated to generate access frequency characteristics, time interval characteristics, and operation type distribution ratios as the initial data access characteristics. Next, the system performs the first data analysis processing on the initial data access characteristics, using a classification model trained on access samples labeled with data popularity categories over the past two years to predict the access characteristics and popularity trends of the current raw data. For example, the model identifies frequently accessed customer repayment records and active transaction user behavior logs from the past week as "hot data," while scanned copies of existing loan contracts that have not been accessed for a long time are considered "cold data." Based on the initial heat status of this data, the system queries the defined storage device performance-heat mapping relationship (e.g., NVMe SSDs for hot data, hybrid HDDs for warm data, and SATA HDDs for cold data), and matches storage devices that meet the performance requirements from the device resource pool. Considering both device read / write performance and storage cost, the system prioritizes low-latency, high-IOPS devices as the primary target storage device for hot data, writing the raw data to the target device via an SSL channel and simultaneously recording the mapping relationship in the global metadata index. As the system runs, the monitoring module dynamically sets a monitoring window (e.g., 7 days), records read / write events for each data block on the target device, and statistically analyzes its access frequency and operation interval in real time, comparing them with historical characteristics to generate access trend change factors and subsequent access characteristics. In the second data analysis and processing, the system uses a time series model to analyze the changing trend of data access patterns, predicts the access fluctuation range of the future window, and calculates the deviation degree in conjunction with the threshold range of the original heat status. When the deviation degree significantly exceeds the preset range, and the current device load is detected to be high or the data has exceeded the set storage period, a heat status change judgment and migration condition judgment are triggered. The system then selects a new target storage device based on the changed popularity status. For example, repayment log files that are determined to have decreased popularity are migrated from a high-performance SSD to a capacity HDD. During the migration process, a complete data synchronization is performed through a secure channel. Subsequently, the original copy is deleted, the global index is updated, and data query and task scheduling strategies are adjusted in real time according to the performance of the target device, thus realizing intelligent popularity management of financial business data.

[0191] In the healthcare sector, a regional medical center's data platform needs to handle massive amounts of patient medical records, medical images, laboratory reports, and follow-up records. Faced with data growth and storage resource constraints, the center deployed an intelligent data platform with AI-driven capabilities for identifying and migrating hot and cold data. The system first collects raw data, such as recently uploaded CT images, historical hospitalization records, and patient medication logs. Within a set timeframe, it extracts access logs, calculates the number of accesses per unit time, the average interval between adjacent accesses, and the distribution of operation types, generating a complete initial data access characterization. The platform uses an AI-trained heat classification model to perform pattern recognition on these data access characteristics, identifying CT data in continuous image viewing requests as "hot data" and long-unaccessed electronic medical records as "cold data," generating corresponding initial heat statuses. Based on these heat statuses, the system selects suitable devices from a multi-level storage device pool: for example, high-speed image servers for hot CT images and ordinary NAS storage devices for cold medical record documents. After comprehensive weighting, the first target storage device is selected, and the secure writing and mapping of data is completed. During operation, the system continuously monitors data access behavior, setting monitoring cycles based on popularity categories (e.g., daily monitoring for emergency-related images, weekly monitoring for follow-up reports), and generating real-time access frequency indicators and trend change factors. During analysis, the platform uses historical patient behavior models and access time-series models to predict access trends for the next period and determine if a shift in popularity has occurred. If access to a recent batch of CT data decreases and the system detects excessive load on the current image server, the system will determine that migration conditions are met. At this point, the platform migrates this CT data to a mid-range HDD storage array, updates the mapping information to the index database, and simultaneously adjusts the access routing configuration of the doctor's workstation to ensure timely access without consuming high-speed resources. This series of operations not only achieves intelligent management of the patient data lifecycle but also greatly alleviates the storage resource pressure faced by medical institutions while ensuring the timeliness of medical services.

[0192] This embodiment utilizes a data migration mechanism driven by changes in data popularity to dynamically redistribute storage for data at different lifecycle stages. This allows hot data to be matched with high-performance devices and cold data to be migrated to low-cost devices, significantly improving resource utilization efficiency. By introducing load rate and residency period as criteria, unnecessary data migration triggered by a single change in popularity is avoided, enhancing system stability and cost-effectiveness. Metadata indexing and secure channel management enable precise location and highly secure transmission during the data migration process, preventing data consistency or information leakage issues caused by migration operations. Simultaneously, by updating access routing policies, the system's read / write paths are adjusted in real-time after migration, ensuring seamless logical switching of access requests to the new target device, improving the overall system's adaptability and scalability.

[0193] In one embodiment, a data hierarchical storage device based on access representation is provided, which corresponds one-to-one with the data hierarchical storage method based on access representation described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data hierarchical storage device based on access representation of the present invention. The modules include an access representation acquisition module 10, an access characteristic identification module 20, an initial write decision module 30, an access behavior monitoring module 40, a popularity status determination module 50, and a storage migration execution module 60. Detailed descriptions of each functional module are as follows:

[0194] Access representation acquisition module 10 is used to acquire the initial data access representation of the raw data;

[0195] The access characteristic identification module 20 is used to identify the data access characteristics of the original data based on the initial data access characterization and to generate the initial data heat status of the original data through the first data analysis and processing.

[0196] The initial write decision module 30 is used to allocate a first target storage device to the original data according to the initial data heat status and the preset storage device performance correspondence, and store the original data into the first target storage device;

[0197] Access behavior monitoring module 40 is used to monitor subsequent data access characteristics of the data already stored in the first target storage device;

[0198] The heat status determination module 50 is used to determine, based on the subsequent data access characterization, whether the heat status of the stored data has changed compared to the initial data heat status through a second data analysis process.

[0199] The storage migration execution module 60 is used to allocate a second target storage device to the stored data and move the stored data from the first target storage device to the second target storage device when the data popularity status of the stored data changes and the preset data movement conditions are met.

[0200] In one embodiment, the access characterization acquisition module 10 is specifically used for:

[0201] Collect data access logs of raw data within a preset time period, the data access logs including the access operation types of the raw data;

[0202] Based on the data access log, the total number of times the original data was accessed within a unit time window is determined, and access frequency characteristics are generated.

[0203] Based on the data access log, the average time difference between two consecutive access operations of the original data is determined, and a time interval feature is generated;

[0204] By integrating the access frequency characteristics, the time interval characteristics, and the distribution ratio of the access operation types, an initial data access representation of the original data is generated.

[0205] In one embodiment, the access characteristic identification module 20 is specifically used for:

[0206] The initial data access representation is preprocessed to generate a preprocessed initial data access representation;

[0207] The preprocessed initial data access representation is input into a classification model trained based on historical data access representations and corresponding data popularity status labels.

[0208] The classification model is used to perform pattern matching on the access characteristics of the original data to generate the classification probability distribution of the original data.

[0209] Based on the classification probability distribution, the original data is divided into one of the categories of hot data, warm data, or cold data to generate the initial data heat status;

[0210] The initial data popularity status is associated with the metadata description file of the original data in the form of a unique identifier.

[0211] In one embodiment, the initial writing decision module 30 is specifically used for:

[0212] Based on the initial data heat status, query the preset storage device resource pool information to match the corresponding storage device performance requirements, and filter out a set of candidate storage devices that meet the storage device performance requirements from the storage device resource pool information;

[0213] A comprehensive priority score is determined based on the read / write performance indicators and unit storage cost of each storage device in the candidate storage device set.

[0214] Select the storage device with the highest overall priority score as the first target storage device;

[0215] Establish a secure data transmission channel with the first target storage device, and transmit the original data to the storage space of the first target storage device through the secure data transmission channel;

[0216] When the original data is stored in the first target storage device, the global metadata index is updated to record the storage mapping relationship between the original data and the first target storage device.

[0217] In one embodiment, the access behavior monitoring module 40 is specifically used for:

[0218] For the data already stored in the first target storage device, a dynamic monitoring time window is set that is dynamically adjusted according to the initial data heat status;

[0219] Within the dynamic monitoring time window, capture every read operation event and write operation event of the stored data to generate a real-time access operation record.

[0220] The number of read operation events and write operation events of the stored data within the dynamic monitoring time window is counted to generate a dynamic access frequency index for the current monitoring window.

[0221] The average time difference between two consecutive access operations of the stored data within the dynamic monitoring time window is determined, and a dynamic time interval index for the current monitoring window is generated.

[0222] Analyze the difference between the dynamic access frequency index and the historical access frequency characteristics to generate an access trend change factor;

[0223] The dynamic access frequency index, the dynamic time interval index, and the access trend change factor are integrated to generate subsequent data access characterization.

[0224] In one embodiment, the heat status determination module 50 is specifically used for:

[0225] The subsequent data access representation is standardized to generate a standardized subsequent data access representation.

[0226] The standardized subsequent data access representation is input into a time series analysis model trained based on historical data access representation time series and corresponding data popularity status change labels.

[0227] The time series analysis model is used to generate the predicted access frequency fluctuation range and the predicted time interval fluctuation range of the stored data within a future preset time window.

[0228] The predicted access frequency fluctuation range is compared with the baseline access frequency threshold corresponding to the initial data popularity state to generate a frequency fluctuation deviation.

[0229] The predicted time interval fluctuation range is compared with the baseline time interval threshold corresponding to the initial data heat state to generate the interval fluctuation deviation.

[0230] Based on the weighted sum of the frequency fluctuation deviation and the interval fluctuation deviation, it is determined whether the data popularity status of the stored data has changed;

[0231] When the weighted sum exceeds a preset deviation threshold, it is determined that the data popularity status has changed, and a data popularity status change instruction is generated. The data popularity status change instruction is then associated with the metadata description file of the stored data.

[0232] In one embodiment, the storage migration execution module 60 is specifically used for:

[0233] When the data popularity status of the stored data changes, obtain the data popularity status change instruction and the associated metadata description file of the stored data;

[0234] Based on the storage mapping relationship of the first target storage device recorded in the metadata description file, verify whether the preset data movement conditions are met;

[0235] When the preset data movement conditions are met, according to the target data heat state indicated in the data heat state change instruction, a second target storage device that matches the target data heat state is selected from the storage device resource pool information;

[0236] Send a data migration instruction to the first target storage device, and transfer the stored data in the first target storage device to the storage space of the second target storage device through a secure data transmission channel;

[0237] After the migration of the stored data is completed, the stored data in the first target storage device is deleted, and the storage mapping relationship in the global metadata index is updated to record the mapping information between the stored data and the second target storage device.

[0238] Based on the read / write performance metrics of the second target storage device, adjust the routing strategy for subsequent data access requests.

[0239] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements server-side functions or steps of a data hierarchical storage method based on access representation.

[0240] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements user-side functions or steps of a data hierarchical storage method based on access representation.

[0241] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0242] Obtain the initial data access representation of the raw data;

[0243] Based on the initial data access characterization, the data access characteristics of the original data are identified through the first data analysis and processing to generate the initial data popularity status of the original data.

[0244] Based on the initial data popularity status and the preset storage device performance correspondence, a first target storage device is allocated to the original data, and the original data is stored in the first target storage device;

[0245] Monitor subsequent data access characteristics of the data already stored in the first target storage device;

[0246] Based on the subsequent data access characterization, the second data analysis and processing determines whether the data popularity status of the stored data has changed compared to the initial data popularity status;

[0247] When the data popularity status of the stored data changes and the preset data movement conditions are met, a second target storage device is allocated to the stored data, and the stored data is moved from the first target storage device to the second target storage device.

[0248] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0249] Obtain the initial data access representation of the raw data;

[0250] Based on the initial data access characterization, the data access characteristics of the original data are identified through the first data analysis and processing to generate the initial data popularity status of the original data.

[0251] Based on the initial data popularity status and the preset storage device performance correspondence, a first target storage device is allocated to the original data, and the original data is stored in the first target storage device;

[0252] Monitor subsequent data access characteristics of the data already stored in the first target storage device;

[0253] Based on the subsequent data access characterization, the second data analysis and processing determines whether the data popularity status of the stored data has changed compared to the initial data popularity status;

[0254] When the data popularity status of the stored data changes and the preset data movement conditions are met, a second target storage device is allocated to the stored data, and the stored data is moved from the first target storage device to the second target storage device.

[0255] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0256] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0257] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0258] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data hierarchical storage method based on access representation, characterized in that, Includes the following steps: Obtain the initial data access representation of the raw data; Based on the initial data access characterization, the data access characteristics of the original data are identified through the first data analysis and processing to generate the initial data popularity status of the original data. Based on the initial data popularity status and the preset storage device performance correspondence, a first target storage device is allocated to the original data, and the original data is stored in the first target storage device; Monitor subsequent data access characteristics of the data already stored in the first target storage device; Based on the subsequent data access representation, a second data analysis process is used to determine whether the data popularity status of the stored data has changed compared to the initial data popularity status. This includes: standardizing the subsequent data access representation to generate a standardized subsequent data access representation; inputting the standardized subsequent data access representation into a time series analysis model trained based on historical data access representation time series and corresponding data popularity status change labels; using the time series analysis model to generate the predicted access frequency fluctuation range and predicted time interval fluctuation range of the stored data within a future preset time window; and then processing the predicted access frequency fluctuation range into the time series analysis model. The frequency fluctuation range is compared with the baseline access frequency threshold corresponding to the initial data popularity state to generate a frequency fluctuation deviation; the predicted time interval fluctuation range is compared with the baseline time interval threshold corresponding to the initial data popularity state to generate an interval fluctuation deviation; based on the weighted sum of the frequency fluctuation deviation and the interval fluctuation deviation, it is determined whether the data popularity state of the stored data has changed; when the weighted sum exceeds a preset deviation threshold, it is determined that the data popularity state has changed, and a data popularity state change instruction is generated, and the data popularity state change instruction is associated with the metadata description file of the stored data; When the data popularity status of the stored data changes and the preset data movement conditions are met, a second target storage device is allocated to the stored data, and the stored data is moved from the first target storage device to the second target storage device.

2. The data hierarchical storage method based on access representation as described in claim 1, characterized in that, The initial data access representation of the raw data is obtained, including: Collect data access logs of raw data within a preset time period, the data access logs including the access operation types of the raw data; Based on the data access log, the total number of times the original data was accessed within a unit time window is determined, and access frequency characteristics are generated. Based on the data access log, the average time difference between two consecutive access operations of the original data is determined, and a time interval feature is generated; By integrating the access frequency characteristics, the time interval characteristics, and the distribution ratio of the access operation types, an initial data access representation of the original data is generated.

3. The data hierarchical storage method based on access representation as described in claim 1, characterized in that, Based on the initial data access characterization, the original data is analyzed and its data access characteristics are identified through a first data analysis process to generate an initial data popularity status for the original data, including: The initial data access representation is preprocessed to generate a preprocessed initial data access representation; The preprocessed initial data access representation is input into a classification model trained based on historical data access representations and corresponding data popularity status labels. The classification model is used to perform pattern matching on the access characteristics of the original data to generate the classification probability distribution of the original data. Based on the classification probability distribution, the original data is divided into one of the categories of hot data, warm data, or cold data to generate the initial data heat status; The initial data popularity status is associated with the metadata description file of the original data in the form of a unique identifier.

4. The data hierarchical storage method based on access representation as described in claim 1, characterized in that, Based on the initial data popularity status and the preset storage device performance correspondence, a first target storage device is allocated to the original data, and the original data is stored in the first target storage device, including: Based on the initial data heat status, query the preset storage device resource pool information to match the corresponding storage device performance requirements, and filter out a set of candidate storage devices that meet the storage device performance requirements from the storage device resource pool information; A comprehensive priority score is determined based on the read / write performance indicators and unit storage cost of each storage device in the candidate storage device set. Select the storage device with the highest overall priority score as the first target storage device; Establish a secure data transmission channel with the first target storage device, and transmit the original data to the storage space of the first target storage device through the secure data transmission channel; When the original data is stored in the first target storage device, the global metadata index is updated to record the storage mapping relationship between the original data and the first target storage device.

5. The data hierarchical storage method based on access representation as described in claim 1, characterized in that, Monitoring subsequent data access characteristics of the data already stored in the first target storage device includes: For the data already stored in the first target storage device, a dynamic monitoring time window is set that is dynamically adjusted according to the initial data heat status; Within the dynamic monitoring time window, capture every read operation event and write operation event of the stored data to generate a real-time access operation record. The number of read operation events and write operation events of the stored data within the dynamic monitoring time window is counted to generate a dynamic access frequency index for the current monitoring window. The average time difference between two consecutive access operations of the stored data within the dynamic monitoring time window is determined, and a dynamic time interval index for the current monitoring window is generated. Analyze the difference between the dynamic access frequency index and the historical access frequency characteristics to generate an access trend change factor; The dynamic access frequency index, the dynamic time interval index, and the access trend change factor are integrated to generate subsequent data access characterization.

6. The data hierarchical storage method based on access representation as described in claim 1, characterized in that, When the data popularity status of the stored data changes and a preset data movement condition is met, a second target storage device is allocated to the stored data, and the stored data is moved from the first target storage device to the second target storage device, including: When the data popularity status of the stored data changes, obtain the data popularity status change instruction and the associated metadata description file of the stored data; Based on the storage mapping relationship of the first target storage device recorded in the metadata description file, verify whether the preset data movement conditions are met; When the preset data movement conditions are met, according to the target data heat state indicated in the data heat state change instruction, a second target storage device that matches the target data heat state is selected from the storage device resource pool information; Send a data migration instruction to the first target storage device, and transfer the stored data in the first target storage device to the storage space of the second target storage device through a secure data transmission channel; After the migration of the stored data is completed, the stored data in the first target storage device is deleted, and the storage mapping relationship in the global metadata index is updated to record the mapping information between the stored data and the second target storage device. Based on the read / write performance metrics of the second target storage device, adjust the routing strategy for subsequent data access requests.

7. A data hierarchical storage device based on access representation, characterized in that, The data hierarchical storage device based on access representation includes: The access representation acquisition module is used to obtain the initial data access representation of the raw data; The access characteristic identification module is used to identify the data access characteristics of the original data based on the initial data access characterization and to generate the initial data popularity status of the original data through the first data analysis and processing. The initial write decision module is used to allocate a first target storage device to the original data according to the initial data heat status and the preset storage device performance correspondence, and store the original data into the first target storage device; The access behavior monitoring module is used to monitor subsequent data access characteristics of the data already stored in the first target storage device; The popularity status determination module is used to determine, based on the subsequent data access representation, whether the popularity status of the stored data has changed compared to the initial data popularity status through a second data analysis process. This includes: standardizing the subsequent data access representation to generate a standardized subsequent data access representation; inputting the standardized subsequent data access representation into a time series analysis model trained based on historical data access representation time series and corresponding data popularity status change labels; using the time series analysis model to generate the predicted access frequency fluctuation range and predicted time interval fluctuation range of the stored data within a future preset time window; and then... The predicted access frequency fluctuation range is compared with the baseline access frequency threshold corresponding to the initial data popularity state to generate a frequency fluctuation deviation; the predicted time interval fluctuation range is compared with the baseline time interval threshold corresponding to the initial data popularity state to generate an interval fluctuation deviation; based on the weighted sum of the frequency fluctuation deviation and the interval fluctuation deviation, it is determined whether the data popularity state of the stored data has changed; when the weighted sum exceeds a preset deviation threshold, it is determined that the data popularity state has changed, and a data popularity state change instruction is generated, and the data popularity state change instruction is associated with the metadata description file of the stored data; The storage migration execution module is used to allocate a second target storage device to the stored data and move the stored data from the first target storage device to the second target storage device when the data popularity status of the stored data changes and the preset data movement conditions are met.

8. A computer device, characterized in that, The computer device includes a memory, a processor, and an access-based hierarchical data storage program stored on the memory and executable on the processor, wherein the access-based hierarchical data storage program, when executed by the processor, implements the steps of the access-based hierarchical data storage method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a data hierarchical storage program based on access representation, which, when executed by a processor, implements the steps of the data hierarchical storage method based on access representation as described in any one of claims 1-6.

Images