A method for logical layering and data scheduling of a magneto-optical integrated storage device

Abstract

The present application relates to a kind of logic layering and data scheduling method of magneto-optical integrated storage device, comprising: multiple modal files are split into multiple logical objects according to semantics, then cut into multiple physical blocks;Based on the comprehensive data temperature value of physical block calculated by multidimensional feature, and accordingly, physical block is divided into hot data, warm data or cold data;Multiple heterogeneous storage media in magneto-optical integrated storage device are combined according to the preset storage medium proportion to construct at least the logical storage pool including high frequency, medium frequency and low frequency storage pool, and unified management is carried out by the same central processing unit through polling management layer;Physical block is initially allocated to corresponding high frequency, medium frequency or low frequency storage pool, and the physical block in each storage pool is allocated to the specific storage medium contained in the storage pool;The comprehensive data temperature value of each physical block is periodically recalculated, and physical block is dynamically migrated between different storage pools according to its change.This method can realize the optimization balance of storage performance and cost.

Description

Technical Field

[0001] This invention relates to the field of storage technology, specifically to a logical layering and data scheduling method for a magneto-optical integrated storage device. Background Technology

[0002] With the deepening of digital transformation, the total amount of global data is exploding at an unprecedented rate. This data exhibits different access characteristics depending on its stage of lifecycle. Statistics show that in typical big data application scenarios, data requiring frequent access accounts for approximately 5% of the total data volume, data with the next highest access frequency accounts for approximately 15%, while the vast majority (approximately 80%) is data with extremely low access frequency.

[0003] Traditional single-media storage solutions struggle to meet these complex storage demands. While all-flash arrays (all-SSDs) offer superior performance, their high cost makes them unsuitable for storing massive amounts of cold data; while high-capacity hard disk drives (HDDs) or tape libraries, though less expensive, suffer from access latency and I / O performance that cannot meet the real-time requirements of hot data.

[0004] To strike a balance between storage cost and access performance, tiered storage emerged. Its core idea is to store data of varying frequency of use on storage media with different costs and performance levels. Magneto-optical-electrical integrated storage systems combine electrical storage (high performance, high cost) represented by solid-state drives (SSDs), magnetic storage (mid-range performance and cost) represented by hard disk drives (HDDs), and optical storage (low cost, long lifespan, high security) represented by Blu-ray disc libraries, providing an ideal physical foundation for tiered storage.

[0005] Several tiered storage solutions already exist. These solutions typically determine data's access frequency to categorize it as hot, warm, or cold. Hot data is stored on SSDs (Simplified Chinese SSDs) for faster read / write speeds, warm data on HDDs (High-Speed ​​Digital Drives), and cold data on Blu-ray discs. This tiered storage approach can balance data access efficiency and storage cost to some extent. However, existing tiered storage solutions generally suffer from the following problems:

[0006] 1. Limited Dimensions for Data Popularity Segmentation: Most traditional solutions rely primarily on data access frequency or recent access time to classify data as hot or cold. This approach ignores the inherent attributes of the data (such as data modality) and the business value it carries (such as business characteristics), resulting in inaccurate segmentation. Some infrequently accessed but highly critical business data may be incorrectly degraded to cold storage, leading to significant access delays at critical moments.

[0007] 2. Overly coarse data granularity: Current conventional methods for classifying data into hot, warm, and cold categories typically label multimodal unstructured data such as video, audio, and documents at the file level. This classification of data categories is rather coarse. When only part of the data in a file needs to be retrieved, the entire file must be retrieved from the storage resource pool, which is both time-consuming and energy-intensive.

[0008] 3. The pooling solution for magneto-optical integrated storage is not convenient enough: Existing magneto-optical hybrid storage devices combine multiple storage media such as SSDs, HDDs, and Blu-ray. For business systems, it may be necessary to use multiple magneto-optical integrated storage devices across data centers or even across server rooms. Currently, most pooling solutions for multiple devices are divided according to media type, such as putting SSDs together in a hot data pool, HDDs together in a warm data pool, and Blu-ray storage together in a cold data pool. However, the current pooling solution is too simple and cannot meet some business characteristics. For example, some data, although cold data, may be accessed frequently at any time. Business systems want the cold data pool to also have some high-performance SSDs to meet the needs of such business operations.

[0009] 4. Data storage strategies are not intelligent enough: Existing data storage strategies are too simplistic. For example, if a warm data resource pool has multiple different storage devices, the storage of a certain warm data will arbitrarily choose one of the storage devices. There is a lack of algorithms for accurate calculation and planning, which will lead to uneven device load, with some devices being overloaded, resulting in waste of resources and energy. Summary of the Invention

[0010] The purpose of this invention is to provide a logical layering and data scheduling method for a magneto-optical integrated storage device, so as to solve the problems existing in the prior art and achieve an optimized balance between storage performance and cost.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is: a logical layering and data scheduling method for a magneto-optical integrated storage device, characterized in that it includes:

[0012] Multimodal files are semantically split into multiple logical objects, and each logical object is further divided into multiple physical blocks. The comprehensive data temperature value of each physical block is calculated based on multidimensional features, and the physical block is divided into hot data, warm data, or cold data according to the comprehensive data temperature value, which serves as its temperature label. A data index containing access ID, block size, block location, temperature label, and semantic embedding is established for each physical block.

[0013] The various heterogeneous storage media in the magneto-optical integrated storage device are combined according to a preset storage media ratio to construct a logical storage pool that includes at least a high-frequency storage pool, a medium-frequency storage pool, and a low-frequency storage pool; the logical storage pool is uniformly managed by the same central processing unit through a polling management layer;

[0014] Based on the temperature label of the physical block, the physical block is initially allocated to the corresponding high-frequency, medium-frequency or low-frequency storage pool, and the physical blocks in each storage pool are allocated to the specific storage medium contained in that storage pool; the comprehensive data temperature value of each physical block is periodically recalculated, and the physical blocks are dynamically migrated between different storage pools according to the change of the comprehensive data temperature value.

[0015] Furthermore, the step of semantically splitting the multimodal file into multiple logical objects, and then further dividing each logical object into multiple physical blocks, specifically includes:

[0016] The multimodal files include video files, audio files, and document files. For video files, a scene detection algorithm based on content similarity is used for segmentation, identifying scene boundaries in the video and dividing the video into multiple video logical objects based on these boundaries. For audio files, a speech activity detection algorithm is used for segmentation, dividing the audio into multiple audio logical objects based on silence segment boundaries. For document files, the chapter or page structure of different document formats is analyzed, and the document is divided into multiple document logical objects based on the chapter or page structure.

[0017] Set a fixed segmentation threshold. For logical objects whose size exceeds the segmentation threshold, they are segmented into multiple physical blocks according to the segmentation threshold. The segmentation thresholds for video logical objects, audio logical objects, and document logical objects are set independently.

[0018] Furthermore, the multidimensional features include at least access frequency features based on physical block access history, data modality heat features based on the file modality to which the physical block belongs and the access heat of that file modality, and business characteristic features based on physical block service business.

[0019] The comprehensive data temperature value of each physical block is calculated by combining access frequency characteristics, data modality heat characteristics, and business characteristic characteristics. Then, the comprehensive data temperature value is compared with the preset temperature threshold to classify the physical block into hot data, warm data, or cold data.

[0020] Furthermore, the method for calculating the comprehensive data temperature value is as follows:

[0021]

[0022] in, For the comprehensive data temperature value, For access frequency score, The data modality heat score, Assigning scores based on business characteristics. , , These are the weighting factors for the access frequency score, data modality popularity score, and business characteristic score, respectively. ;

[0023] The access frequency score is calculated based on the access history of the physical block by introducing a time decay effect. The calculation formula is as follows:

[0024]

[0025] in, The current access time. The last access time, This represents the current access frequency score for the physical block. This is the access frequency score calculated based on the last access to the physical block. The time interval between the current access and the previous access. Here, A is the time decay coefficient, and A is the gain in popularity per visit.

[0026] The data modality heat score is calculated based on the file modality to which the physical block belongs and the access heat of that file modality. The calculation formula is as follows:

[0027]

[0028] in, This represents the current data modality heat score for the physical block. This is the modal heat score of the data calculated during the last access to the physical block. Modal influence factor, The rate of increase in the popularity score. for The number of times the file mode to which the physical block belongs is accessed within the time interval. The rate at which the heat score decreases;

[0029] The business characteristic score is calculated based on a weighted average of data source, service level agreement, and regulatory compliance, using the following formula:

[0030]

[0031] in, Score based on data source. For service level agreement scores, The score represents regulatory compliance, and N(.) is the normalization function. , , These are the weighting coefficients;

[0032] Two preset dynamic thresholds T hot and T warm The physical blocks are divided into data temperature partitions as follows:

[0033] If CDT > T hot Then the physical block is divided into hot data;

[0034] If T warm < CDT ≤ T hot Then the physical block will be divided into temperature data;

[0035] If CDT ≤ T warm Then the physical block will be divided into cold data.

[0036] Furthermore, the structure of the data index is as follows:

[0037] {OID:[size,temperature,physicallocation,semantic embedding,checksum]}

[0038] Wherein, OID represents the access ID of the physical block, OID = original file ID + logical object number + physical block sequence number, size represents the size of the physical block, temperature represents the temperature label of the physical block, physical location represents the physical storage location of the physical block, and semantic embedding represents the semantic embedding vector of the physical block.

[0039] Furthermore, the various heterogeneous storage media include at least three different storage media types: solid-state drives (SSDs), hard disk drives (HDDs), and Blu-ray discs.

[0040] The magneto-optical integrated storage device combines SSD, HDD, and Blu-ray disc storage media of three types, and constructs a high-frequency storage pool, a mid-frequency storage pool, and a low-frequency storage pool according to a preset storage media ratio based on the network latency of accessing each storage device. Among them, the high-frequency storage pool uses SSD as the main storage medium and configures a small number of HDDs according to a set ratio; the mid-frequency storage pool uses HDD as the main storage medium and configures some SSDs and Blu-ray discs according to a set ratio; the low-frequency storage pool uses Blu-ray discs and HDDs as the main storage media and configures a small number of SSDs according to a set ratio.

[0041] Furthermore, the method for constructing the high-frequency storage pool is as follows: priority is given to selecting the storage device with the shortest access latency, with SSD as the main storage medium and HDD as the auxiliary storage device, and the capacity of SSD is greater than that of HDD.

[0042] The method for constructing the intermediate frequency storage pool is as follows: after selecting the high frequency storage pool, the storage device with the shortest access latency is selected from the remaining storage devices, with HDD as the main storage medium, and SSD and Blu-ray discs are mixed and configured, with HDD having the largest capacity.

[0043] The method for constructing the low-frequency storage pool is as follows: after selecting the high-frequency storage pool and the medium-frequency storage pool, the storage medium with the shortest access latency is selected from the remaining storage media, with Blu-ray discs and HDDs as the main storage media, and a small number of SSDs configured, with the capacity of Blu-ray discs accounting for a larger proportion than that of HDDs and the capacity of SSDs accounting for the smallest proportion.

[0044] Furthermore, the polling management layer is implemented in the form of a kernel-mode driver and adopts a multi-threaded architecture, including a high-priority real-time thread for polling the solid-state drive, a worker thread for processing mechanical hard drive requests, and a background management thread for managing the status of Blu-ray discs.

[0045] The polling management layer performs the following operations:

[0046] A time-slicing mechanism is used to assign different time-slice weights to storage devices with different storage media types;

[0047] Based on time slice weights, the status of each storage device is periodically polled, including ready, busy, and sleeping.

[0048] Only schedule input / output requests for storage devices in the ready state, and send wake-up commands to storage devices in the hibernation state and suspend the corresponding requests until their state becomes ready;

[0049] The polling management layer is also used to: intercept and parse physical block read / write requests initiated by upper-layer applications, convert them into protocol commands corresponding to the target storage device type, and distribute them to the corresponding storage device queue for processing.

[0050] Furthermore, a graph algorithm is used to allocate physical blocks within each storage pool to the specific storage media contained in that storage pool, specifically including:

[0051] A graph structure is constructed based on the temporal relationship and semantic similarity of physical blocks, wherein there are edges between temporally continuous physical blocks or physical blocks with semantic similarity greater than a preset threshold;

[0052] Feature learning is performed on each physical block node in the graph, and the node's own attributes and the attributes of its first-order neighbor nodes are aggregated to generate a feature vector. The node attributes include semantic embedding vector, access frequency score, data modality heat score, business characteristic score, access frequency and physical size.

[0053] Based on feature vectors, a classification model is used to predict the target storage medium of physical blocks, enabling the allocation of physical blocks on SSDs, HDDs, and Blu-ray discs within the storage pool.

[0054] Furthermore, when the overall data temperature value of a physical block exceeds the upper limit threshold of its storage pool, an upgrade migration is triggered: the physical block is immediately migrated from the current storage pool to a higher-performance storage pool. The migration process is executed asynchronously in the background. After the migration is completed, the data index of the physical block is updated to point to the new storage location.

[0055] When the overall data temperature value of a physical block remains below the lower limit threshold of its storage pool for a preset period, a degradation migration is triggered: the physical block is migrated from the current storage pool to a lower-performance storage pool, and the original storage location space is released and the data index is updated after the migration is completed.

[0056] The system monitors the percentage of used capacity in high-frequency, mid-frequency, and low-frequency storage pools in real time. When the percentage of used capacity in any storage pool exceeds a preset threshold, the storage media are recombined and the storage pools are reconstructed.

[0057] Compared with existing technologies, this invention has the following advantages: By introducing multi-dimensional features to quantify data temperature, this invention achieves more accurate hot and cold data partitioning than simply relying on access frequency; at the same time, through innovative logical storage pool construction and polling management mechanisms, it breaks the fixed binding between physical storage devices and CPUs, realizing a cross-device, on-demand, dynamically mixed media elastic resource pool, greatly improving resource utilization and energy efficiency; based on this, combined with data allocation and migration strategies, it can optimize load balancing and reduce data transfer energy consumption while ensuring performance, thus providing a high-performance, low-cost, and high-energy-efficiency solution for massive multimodal data storage. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the implementation of the logical layering and data scheduling method for the magneto-optical integrated storage device provided in this embodiment of the invention.

[0059] Figure 2 This is a block diagram of a hybrid media logic layered design scheme in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the edge connection relationship between the 6 nodes in an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of an undirected graph containing 6 nodes and 8 edges obtained in an embodiment of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0064] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0065] like Figure 1 As shown, this embodiment provides a logical layering and data scheduling method for a magneto-optical integrated storage device, including three steps: data temperature quantization and partitioning, logical partitioning of the magneto-optical integrated storage device, and data allocation and migration.

[0066] Step S1: Data temperature quantization and segmentation based on multidimensional features: The multimodal file is semantically split into multiple logical objects, and each logical object is then divided into multiple physical blocks; the comprehensive data temperature (CDT) value of each physical block is calculated based on multidimensional features, and the physical block is divided into hot data, warm data, or cold data according to the comprehensive data temperature value, which serves as its temperature label; a data index containing access ID, block size, block location, temperature label, and semantic embedding is established for each physical block.

[0067] The multimodal files include video files, audio files, and document files. The multimodal files are split into multiple logical objects, specifically:

[0068] 1) For video files, a scene detection algorithm based on content similarity is used for segmentation, identifying scene boundaries in the video and segmenting the video into multiple video logical objects based on the scene boundaries;

[0069] 2) For audio files, a speech activity detection algorithm is used to split the audio into multiple audio logical objects based on the boundaries of silent segments;

[0070] 3) For document files, analyze the chapter or page structure of different document formats and split the document into multiple logical document objects according to the chapter or page structure.

[0071] Set a fixed segmentation threshold. For logical objects whose size exceeds the segmentation threshold, they are segmented into multiple physical blocks according to the segmentation threshold. The segmentation thresholds for video logical objects, audio logical objects, and document logical objects are set independently.

[0072] The multidimensional features include at least access frequency features based on the physical block access history, data modality heat features based on the file modality to which the physical block belongs and the access popularity of that file modality, and business characteristic features based on the physical block service. The comprehensive data temperature value for each physical block is calculated by combining the access frequency features, data modality heat features, and business characteristic features; then, the comprehensive data temperature value is compared with a preset temperature threshold to classify the physical block into hot data, warm data, or cold data.

[0073] The method for calculating the comprehensive data temperature value is as follows:

[0074]

[0075] in, For the comprehensive data temperature value, For access frequency score, The data modality heat score, Assigning scores based on business characteristics. , , These are the weighting factors for the access frequency score, data modality popularity score, and business characteristic score, respectively. .

[0076] The access frequency score is calculated based on the access history of the physical block by introducing a time decay effect. The calculation formula is as follows:

[0077]

[0078] in, The current access time. The last access time, This represents the current access frequency score for the physical block. This is the access frequency score calculated based on the last access to the physical block. The time interval between the current access and the previous access. is the time decay coefficient, and A is the gain in popularity per visit.

[0079] The data modality heat score is calculated based on the file modality to which the physical block belongs and the access heat of that file modality. The calculation formula is as follows:

[0080]

[0081] in, This represents the current data modality heat score for the physical block. This is the modal heat score of the data calculated during the last access to the physical block. Modal influence factor, The rate of increase in the popularity score. for The number of times the file mode to which the physical block belongs is accessed within the time interval. This represents the rate at which the heat score decreases.

[0082] The business characteristic score is calculated based on a weighted average of data source, service level agreement, and regulatory compliance, using the following formula:

[0083]

[0084] in, Score based on data source. For service level agreement scores, The score represents regulatory compliance, and N(.) is the normalization function. , , These are the weighting coefficients.

[0085] Two preset dynamic thresholds T hot and T warm The physical blocks are divided into data temperature partitions as follows:

[0086] If CDT > T hot Then the physical block is divided into hot data;

[0087] If T warm < CDT ≤ T hot Then the physical block will be divided into temperature data;

[0088] If CDT ≤ T warm Then the physical block will be divided into cold data.

[0089] The structure of the data index is as follows:

[0090] {OID:[size,temperature,physicallocation,semantic embedding,checksum]}

[0091] Wherein, OID represents the access ID of the physical block, OID = original file ID + logical object number + physical block sequence number, size represents the size of the physical block, temperature represents the temperature label of the physical block, physical location represents the physical storage location of the physical block, and semantic embedding represents the semantic embedding vector of the physical block.

[0092] S2. Logical partitioning of the magneto-optical integrated storage device: The various heterogeneous storage media in the magneto-optical integrated storage device are combined according to a preset storage media ratio to construct a logical storage pool that includes at least a high-frequency storage pool, a medium-frequency storage pool, and a low-frequency storage pool; the logical storage pool is uniformly managed by the same central processing unit through a polling management layer.

[0093] The various heterogeneous storage media include at least three different storage media types: solid-state drives (SSDs), hard disk drives (HDDs), and Blu-ray discs. During the logical partitioning process, the SSD, HDD, and Blu-ray disc storage devices in the magneto-optical integrated storage device are combined according to a preset storage media ratio to construct a high-frequency storage pool, a mid-frequency storage pool, and a low-frequency storage pool, based on the network latency for accessing each storage device. Specifically, the high-frequency storage pool primarily uses SSDs as the storage medium, with a small number of HDDs configured according to a set ratio; the mid-frequency storage pool primarily uses HDDs as the storage medium, with a portion of SSDs and Blu-ray discs configured according to a set ratio; and the low-frequency storage pool primarily uses Blu-ray discs and HDDs as the storage media, with a small number of SSDs configured according to a set ratio.

[0094] The high-frequency storage pool is constructed as follows: Prioritize storage devices with the shortest access latency, using SSDs as the primary storage medium and HDDs as secondary storage devices, with SSD capacity accounting for a larger proportion than HDD capacity. The mid-frequency storage pool is constructed as follows: After selecting the high-frequency storage pool, prioritize storage devices with the shortest access latency from the remaining storage devices, using HDDs as the primary storage medium, and configuring a mix of SSDs and Blu-ray discs, with HDD capacity accounting for the largest proportion. The low-frequency storage pool is constructed as follows: After selecting the high-frequency and mid-frequency storage pools, prioritize storage media with the shortest access latency from the remaining storage media, using Blu-ray discs and HDDs as the primary storage media, and configuring a small number of SSDs, with Blu-ray disc capacity accounting for a larger proportion than HDD capacity and SSD capacity accounting for the smallest proportion.

[0095] The polling management layer is implemented in the form of a kernel-mode driver and adopts a multi-threaded architecture, including a high-priority real-time thread for polling solid-state drives, a worker thread for processing mechanical hard drive requests, and a background management thread for managing the status of Blu-ray discs.

[0096] The polling management layer performs the following operations:

[0097] A time-slicing mechanism is used to assign different time-slice weights to storage devices with different storage media types;

[0098] Based on time slice weights, the status of each storage device is periodically polled, including ready, busy, and sleeping.

[0099] Input / output requests are scheduled only for storage devices in the ready state. For storage devices in the hibernation state, a wake-up command is sent and the corresponding request is suspended until the device becomes ready.

[0100] The polling management layer is also used to: intercept and parse physical block read / write requests initiated by upper-layer applications, convert them into protocol commands corresponding to the target storage device type, and distribute them to the corresponding storage device queue for processing.

[0101] S3. Data Allocation and Migration: Based on the temperature label of the physical block, the physical block is initially allocated to the corresponding high-frequency, medium-frequency or low-frequency storage pool, and the physical blocks in each storage pool are allocated to the specific storage medium contained in that storage pool; the comprehensive data temperature value of each physical block is periodically recalculated, and the physical blocks are dynamically migrated between different storage pools according to the change of the comprehensive data temperature value.

[0102] During data allocation and migration, a graph algorithm is used to allocate physical blocks within each storage pool to the specific storage media contained in that storage pool, specifically including:

[0103] 1) Construct a graph structure based on the temporal relationship and semantic similarity of physical blocks, where there are edges between temporally continuous physical blocks or physical blocks with semantic similarity greater than a preset threshold;

[0104] 2) Perform feature learning on each physical block node in the graph, and aggregate the node's own attributes and the attributes of its first-order neighbor nodes to generate a feature vector. The node attributes include semantic embedding vector, access frequency score, data modality heat score, business characteristic score, access frequency and physical size.

[0105] 3) Based on feature vectors, the target storage medium of physical blocks is predicted through a classification model, thereby enabling the allocation of physical blocks on SSDs, HDDs, and Blu-ray discs within the storage pool.

[0106] During data allocation and migration, when the overall data temperature value of a physical block exceeds the upper limit threshold of its storage pool, an upgrade migration is triggered: the physical block is immediately migrated from the current storage pool to a higher-performance storage pool. The migration process is executed asynchronously in the background. After the migration is completed, the data index of the physical block is updated to point to the new storage location.

[0107] When the overall data temperature value of a physical block remains below the lower limit threshold of its storage pool for a preset period, a degradation migration is triggered: the physical block is migrated from the current storage pool to a lower-performance storage pool, and the original storage location space is released and the data index is updated after the migration is completed.

[0108] In addition, the system monitors the percentage of used capacity in high-frequency, medium-frequency, and low-frequency storage pools in real time. When the percentage of used capacity in any storage pool exceeds a preset threshold, the storage media are recombined and the storage pools are reconstructed.

[0109] The relevant content of this invention will be further explained below with reference to specific embodiments.

[0110] I. Data thermal quantification and partitioning based on multidimensional features (partitioning of warm, cold, and hot data)

[0111] Currently, the classification of data categories is rather crude. Typically, the entire file is stored in a resource pool based on the frequency of data usage. When a portion of the data in a file needs to be retrieved, the entire file must be retrieved from the storage resource pool and transmitted over a long distance to the local end. This process is time-consuming and energy-intensive. Therefore, it is necessary to further subdivide the files and classify them into warm, cold, and hot data based on the subdivided data.

[0112] Current "whole file" level hot / cold stratification strategies, especially in multimodal and ultra-large file scenarios, can lead to severe "over-transfer" and energy waste. A multimodal file can be broken down from "coarse-grained" to "fine-grained" into several fragments, each labeled with a different data temperature, for subsequent placement on the most suitable storage medium.

[0113] 1. Logical object splitting: The original multimodal file is split into multiple logical objects according to semantic purpose.

[0114] Educational video processing is a typical multimodal, ultra-large file scenario, encompassing various data types such as video, audio, and documents. Taking a university's online course platform as an example, assume the platform has 10TB of teaching files, plus some system data. After format detection based on the data modality, the teaching files are divided into N1 videos (assuming a size of 8TB, N1=10KB), N2 audio files (assuming a size of 1TB, N2=10KB), and N3 documents (assuming a size of 1TB, N3=30KB).

[0115] Based on the file type of the multimodal file, different splitting methods are selected to output logical objects.

[0116] Video Processing: Scene detection is performed using PySceneDetect's detect-content mode. This method, based on content similarity analysis, can identify scene boundaries in the video. When the system detects a teacher switching PPT slides, starting an experiment demonstration, or explaining a new knowledge point, it automatically segments the video into different logical objects. After segmentation, each logical object corresponds to a specific knowledge point or teaching segment. Given N1 video files, the scene detection algorithm segments these N1 files into K1 video logical objects of varying sizes. (The number of logical objects will vary depending on the video; we assume there are K1 video logical objects here.)

[0117] Audio Processing: Voice activity is detected via VAD (Voice Activity Detection), and audio tracks are segmented by silence. When the system detects that the teacher has stopped lecturing or started playing background music, it automatically segments the audio into different logical objects. After segmentation, each logical object corresponds to a segment of lecture content. There are N² teaching audio files, which are segmented into K² audio logical objects of different sizes. (The number of logical objects will vary depending on the audio file; let's assume there are K² audio objects here.)

[0118] Document processing: Parsing the chapter or page structure of different document formats such as Word, PDF, and PPT, and splitting the document into logical objects by chapter or page. For example, splitting chapters such as "Introduction," "Experiment 1," and "Solutions to Exercises" into independent logical objects. For scanned documents, using an OCR engine (such as PaddleOCR) to extract text content and splitting it by paragraph or knowledge point. There are N3 teaching documents, divided into K3 document logical objects of different sizes. (The number of logical objects will vary depending on the document size and format; assuming there are K3 document logical objects here.)

[0119] 2. Physical Blocking: Each multimodal logical object, such as video, audio, and document, is further divided into physical blocks of fixed size.

[0120] Segmentation: Each logical object (video, audio, document, etc.) extracted in the previous step is further segmented. Here, we use a threshold P as the segmentation threshold. If a logical object is smaller than P, no segmentation is performed; if a logical object is larger than P, it is segmented into fixed-size blocks. Different segmentation thresholds P can be selected for videos, audios, and documents. Following the method described above:

[0121] K1 video logical objects will be further divided into M1 video physical chunks (assuming the video size is 8T, the chunking threshold is P=50M, and M1=200K).

[0122] K2 audio logical objects will be further divided into M2 audio physical chunks (assuming the audio size is 1T, the chunking threshold P=10M, and M2=200K).

[0123] The K3 logical document objects will be further divided into M3 physical document chunks (assuming the document size is 1T, the chunking threshold P=5M, and M3=400K).

[0124] Let's assume we get a total of M1+M2+M3=800K physical blocks.

[0125] 3. Semantic Embedding: Different embedding models are used based on the data type to generate semantic embedding vectors for each physical block.

[0126] Video embedding: The CLIP (Contrastive Language-Image Pretraining) model is used to process video chunks and extract 512-dimensional visual semantic vectors.

[0127] Audio embedding: The Whisper model is used to process audio chunks and extract 1024-dimensional speech semantic vectors.

[0128] Text embedding: The BGE model is used to process text chunks and extract 768-dimensional text semantic vectors.

[0129] Vector dimension unification: To solve the problem of inconsistent dimensions of vectors of different modalities, AutoEncoder is used to reduce the dimension and unify the vectors of each modality into a 256-dimensional space.

[0130] 4. Design physical block addressing method: Design a two-level namespace and metadata index for reliable and efficient access to physical blocks.

[0131] Two-level namespace: Naming uses a global object ID (OID) structure as follows:

[0132] OID = Original File ID + Logical Object Number + Physical Block Number.

[0133] Below is an example of a global object ID (OID) for a video chunk:

[0134] Original file ID: video_1 (video file)

[0135] Logical object number: / logic_2

[0136] Physical block number: / chunk_5

[0137] OID: video_0001 / / logic_0002 / chunk0005

[0138] This represents the 6th physical block of the 3rd logical object in the video file video_0001, numbered starting from 0.

[0139] Metadata index: Stored on an SSD, leveraging the SSD's high-speed read / write capabilities to ensure low-latency access to metadata. Metadata is stored as a dictionary with the following structure:

[0140] {OID:[size, temperature, physical location, semantic embedding, checksum]}

[0141] Field description:

[0142] size: The size of the chunk (in bytes). (For example, 3.9M)

[0143] Temperature: The temperature label (hot / warm / cold) is provided by the temperature calculation method below. (Cold data)

[0144] Physical location: The physical storage location of the chunk (e.g., node1:diskA)

[0145] Semantic embedding: Semantic embedding of physical blocks, used for physical block addressing.

[0146] Checksum: Used for data integrity verification (using SHA-256 hash value, such as a1d0c6e83f027327...).

[0147] Here is an example of a metadata index:

[0148] {video_0001 / / logic_0002 / chunk_0005:[3.9M, cold data, node1 / diskA, [semantic vector], a1d0c6e83f027327...]}

[0149] 5. Physical block temperature label construction

[0150] For each physical block chunk, a score is calculated based on the access frequency of that physical block. Calculate a data modal heat score based on the heat and modality of the file containing the physical block. Calculate a service characteristic score based on the characteristics of the service served by the physical block. .

[0151] Taking video_0001 / / logic_0002 / chunk0005 as an example, the access frequency of this physical block is calculated. Based on the popularity of video_0001, the physical block's... Based on which services access the physical block and the characteristics of those services, calculate... .

[0152] Finally, these three scores are weighted together to calculate the Comprehensive Data Temperature (CDT) of a physical block. The calculation method for the Comprehensive Data Temperature is as follows:

[0153]

[0154] in, For comprehensive data temperature, , , These are the weighting factors for access frequency, data modality, and business characteristics, respectively. Their sum is 1, and they can be dynamically adjusted according to the overall business of the system.

[0155] (1) Physical block access frequency score Quantization methods (chunk level)

[0156] The access frequency score is assigned to a physical block. The access frequency score quantization algorithm not only counts the number of accesses but also introduces a time decay effect, meaning that accesses made more recently have a higher weight. The specific calculation method is as follows:

[0157]

[0158] in:

[0159] It is the access frequency score of the physical block in the last calculation.

[0160] It is the time interval between two access events.

[0161] This is the time decay coefficient, used to simulate the natural "cooling" of heat over time. The larger the value of λ, the faster the cooling rate. This parameter is configurable. This reflects the value of "recency": the more recently accessed an item is, the slower its decay occurs; the fewer recently accessed an item is, the faster its decay occurs.

[0162] A represents the single-visit popularity gain, which indicates the increase in popularity brought about by each visit.

[0163] (2) Data modal heat score Quantification methods: (file-level)

[0164] At the file level, the popularity of a physical block is calculated based on the access frequency of the file to which that physical block belongs and the file's modality type, defining a data modality popularity score. The calculation formula is:

[0165]

[0166] in:

[0167] It is the heat score of the file to which the physical block belongs in the last calculation.

[0168] is a modality influence factor. For example, in the scenario of educational documents, K(document) < K(audio) < K(video) can be set to ensure preferential loading of educational videos. Different sizes can also be set according to different business scenarios.

[0169] is the heat rising rate, is the number of visits to the file within a time interval. The more visits within a certain time interval, the more the heat score increases

[0170] is the decreasing rate of heat score. As the time interval decreases according to the rate stably.

[0171] (3) Business characteristic score quantification (business level)

[0172] Calculate the business characteristic score of a physical block at the business level. The calculation is performed according to the importance of the services involved in the physical block. A specific calculation example is as follows:

[0173]

[0174] : Score of data source. For example, core business data gets 4 points, system logs get 3 points, user-generated content gets 2 points, and temporary cache gets 1 point.

[0175] : Service Level Agreement (SLA) score. For example, 99.99% availability (annual downtime ≤ 52 minutes) gets 5 points, 99.9% availability (annual downtime ≤ 8.76 hours) gets 3 points, 99.5% availability (annual downtime ≤ 13.5 hours) gets 2 points.

[0176] : Regulatory compliance score. Compliance with strict regulations such as GDPR / HIPAA gets 6 points, compliance with general compliance requirements gets 3 points, and no special compliance requirements gets 1 point.

[0177] N(.): A normalization function that maps each sub-feature value to the interval [0, 100] to facilitate unified calculation.

[0178] , , : Weights of each sub-feature, preset according to the characteristics of the storage system. For example, = 0.4, = 0.3, = 0.2.

[0179] If a physical block contains core business data, has an SLA of 99.99% availability (annual downtime ≤ 52 minutes), and complies with stringent regulations such as GDPR / HIPAA, then... The calculation result is

[0180] =0.4*40+0.3*50+0.2*60=16+15+12=43

[0181] (4) Data classification

[0182] Calculate the access frequency score for each physical block. Data modal heat score Business characteristic score Next, the overall temperature of the physical block is calculated using weighted averages, and two dynamic thresholds T are set. hot and T warm To divide data based on popularity:

[0183] Hot data: If CDT > T hot The data is then labeled as hot data.

[0184] Temperature data: If T warm < CDT ≤ T hot The data is then labeled as warm data.

[0185] Cold data: If CDT ≤ T warm If so, the data is marked as cold data.

[0186] Taking the data from the aforementioned university online course platform as an example, and following the data temperature calculation method provided in this embodiment, temperature labels are applied to 650K physical blocks. The calculated results are assumed to be as follows:

[0187]

[0188] II. Logical layering of magneto-optical integrated storage devices (high frequency, medium frequency, low frequency)

[0189] 2.1 Construct a flexible resource pool with the same CPU scheduling for heterogeneous SSDs, HDDs, and Blu-ray storage:

[0190] Currently, Blu-ray storage, HDD, and SSD storage devices are often integrated solutions with fixed ratios. For integrated optical-magnetic-electrical storage scenarios, this can easily lead to resource and energy waste. For example, this integrated optical-magnetic-electrical storage unit includes 8TB of Blu-ray storage, 1TB of HDD storage, and 1TB of SSD storage. One business requires 4TB of Blu-ray storage, 2TB of HDD storage, and 1TB of SSD storage. For this business, if a single integrated storage unit is used, there will be an excess of Blu-ray storage but insufficient HDD storage, even though other integrated storage units are located in different physical areas. If another integrated storage unit is allocated to this task, the physical isolation prevents the rapid and flexible allocation of free HDD storage media. Consequently, the remaining Blu-ray storage on the original integrated storage unit cannot be quickly allocated to other tasks, effectively wasting the Blu-ray storage of both integrated storage units.

[0191] Therefore, this invention proposes an innovative solution: logically dividing the storage resources of the magneto-optical integrated storage device to decouple the integrated CPU from the underlying heterogeneous physical storage devices (SSD, HDD, Blu-ray disc library). Blu-ray, HDD, and SSD are combined according to business needs to form high-frequency, mid-frequency, and low-frequency storage pools. Unlike existing technologies, the logical storage pool is dynamically composed of a specific ratio of SSD, HDD, and Blu-ray media, rather than a simple stacking of single media. Note: Existing technology stores a task within an all-in-one machine, which means only the entire rack can be selected. If one rack is insufficient for storage, multiple racks need to be selected for synchronous storage. This means that the ratio of CPU1 to SSD / HDD / Blu-ray storage is fixed and strongly bound. The HDD / SSD / Blu-ray storage managed and bound by CPU1 or CPU2 can only be the hardware within its own all-in-one machine and cannot access HDD / SSD / Blu-ray storage managed by other CPUs. This implies that storage is limited. In contrast, this invention allows CPU1 to simultaneously manage the SSD / HDD / Blu-ray storage previously managed by CPU2. This is equivalent to forming various resource pools with greater flexibility. Furthermore, the SSD / HDD / Blu-ray storage in the resource pool dynamically changes based on the temperature characteristics of the data. This breaks the fixed and strongly bound relationship of hardware resources, enabling flexible resource allocation, on-demand adjustment, and optimal energy efficiency. Since all heterogeneous media are managed uniformly by the same CPU through a polling management layer, when data changes from one activity level to another during its lifecycle (such as from hot to cold), there is no need to migrate across networks between physical resource pools managed by different CPUs. Instead, the metadata pointers need to be updated within the system, which greatly reduces the energy consumption of data transfer.

[0192] As can be seen, the original technology only managed the SSDs, HDDs, and Blu-ray storage media vertically within the storage appliance's CPU1, CPU2, and CPU3. This invention patent enables CPU1 to manage and schedule the SSDs, HDDs, and Blu-rays connected to CPU2 and CPU3. High-frequency, mid-frequency, and low-frequency resource pools were also designed simultaneously.

[0193] In this embodiment, the hybrid media logic layering design scheme is as follows: Figure 2 As shown.

[0194] High-frequency storage pool: Designed to provide extremely low latency and high IOPS. This layer mainly consists of SSDs, supplemented by a small number of HDDs.

[0195] Mid-frequency storage pool: The design goal is to achieve a balance between performance, capacity, and cost. This layer of devices is primarily HDD, but also includes a mix of SSDs, HDDs, and Blu-ray media in certain proportions.

[0196] Low-frequency storage pool: Designed to provide massive, low-cost archival storage. This layer primarily consists of Blu-ray media and HDDs, with a small number of SSDs.

[0197] For a business system, the business server is an independent device, and network connections are required to reach different storage appliances. The network latency to reach different storage appliances varies. Therefore, when dividing the storage pool for each logical level, the issue of minimizing latency needs to be considered.

[0198] Specific hardware implementation plan for CPU decoupling of SSD / HDD / Blu-ray storage media:

[0199] SSDs, HDDs, and Blu-ray storage devices have different interfaces, protocols, and speeds that vary greatly (milliseconds vs. seconds vs. tens of seconds). Traditional storage pools (such as Ceph and vSAN) can only be tiered by storage device type and cannot be mixed in one pool. That is, multiple SSDs form one resource pool, multiple HDDs form one resource pool, and multiple Blu-ray storage devices form one resource pool.

[0200] The core invention of this patent is to unify the pooling management of SSDs, HDDs, and Blu-ray storage. It places SSDs, HDDs, and Blu-ray storage devices from different integrated storage units (originally managed by different CPUs) into a pool for unified management by a single CPU. However, a single CPU cannot simultaneously manage both the "millisecond-response" SSDs and the "ten-second wake-up" Blu-rays, and different storage devices have different protocols. Therefore, this patent uses "time-slice abstraction + asynchronous completion notification" for the CPU. This allows the same CPU to work on "currently responsive storage devices," without blocking or waiting for "unwakeable storage devices," but instead suspending requests and switching time slices.

[0201] 1. A new polling management layer is added between each CPU and the storage devices (SSD, HDD, Blu-ray storage). This polling management layer can interconnect with the interfaces and protocols of the SSDs, HDDs, and Blu-ray storage devices. The polling management layer is implemented as a kernel-mode driver and loaded on the host CPU. This driver accesses the hardware through the hardware interface provided by the operating system. This polling management layer uses a multi-threaded architecture.

[0202] Includes a high-priority real-time thread: dedicated to polling and processing the I / O queues of SSD devices;

[0203] One or more regular worker threads: used to handle HDD I / O requests;

[0204] A background management thread: polls the status of the Blu-ray device at a low frequency, is responsible for sending wake-up commands, and submits the corresponding I / O requests to worker threads for processing after the device is ready.

[0205] Asynchronous I / O Framework: The driver is built entirely on an asynchronous I / O model. Each I / O request submitted by the upper layer is associated with a callback function. The driver places all requests into a lock-free queue, and each worker thread retrieves tasks from the queue, executes them, and calls the callback to notify the upper layer upon completion. Unified Address Abstraction: During initialization, the driver registers a unified block device node (such as / dev / universal-pool) with the upper layer using modern block device frameworks such as blk_mq. Internally, the driver maintains a radix tree or hash table to manage the mapping relationship between logical addresses and (storage device type, physical address).

[0206] 2. Each type of storage device (SSD, HDD, Blu-ray storage device) reports its status in real time to the newly added polling management layer: specific statuses include READY, WAKING, SLEEPING, and BUSY. The CPU only schedules the device when it is in the READY state; other states are skipped directly without waiting.

[0207] 3. The newly added polling management layer assigns time slice weights to each different storage device (SSD:HDD:Blu-ray = 1:4:8), meaning that SSD protocol tasks are processed every 1ms, HDD protocol tasks every 4ms, and Blu-ray protocol tasks every 8ms. The polling management layer polls all storage devices, notifying them of new storage tasks that require protocol processing.

[0208] 4. A new polling management layer identifies and processes only storage requests in the READY state. For example, if the 8ms polling layer encounters Blu-ray storage in the SLEEPING state, it skips the Blu-ray and continues processing SSD tasks for 1ms and HDD storage tasks for 4ms. For storage devices currently in the SLEEPING state, it triggers a wake-up call, waiting until the Blu-ray storage wakes up from the SLEEPING state in the next cycle before performing storage operations and protocol integration.

[0209] 5. The CPU continuously polls different storage devices. Different storage devices notify the CPU through MSI-X interrupts or shared memory queues. The CPU processes the callback when polling in the next time slice.

[0210] 6. In this way, users at the upper layer will not be aware that one CPU manages multiple different heterogeneous storage devices. By adding a polling management layer between the CPU and the storage devices (SSD, HDD, Blu-ray storage devices) and using time-division multiplexing, unified management of different storage devices can be achieved.

[0211] These SSDs, HDDs, and Blu-ray storage devices are all placed in a pool and managed by a single CPU, which avoids data being moved back and forth between different CPU servers, saving time and energy.

[0212]

[0213] 7. In addition to time-slicing management, the polling management layer also needs to intercept and parse the protocols of different storage devices. This allows the CPU to seamlessly handle the protocols of different storage devices without requiring users to manually select whether the storage is on an SSD, HDD, or Blu-ray. The specific interception and parsing process is as follows:

[0214] The newly added polling management layer periodically sends identification commands (NVMeIdentify for SSDs, ATA IDENTIFY DEVICE for HDDs, and ATA Packet Interface for Blu-ray storage) to the storage devices. After receiving the data returned by the storage devices, the polling management layer parses out the storage device's WWN / serial number / media type / capacity / performance parameters and generates a Device-Signature.

[0215] The newly added polling management layer unifies the protocols of various storage devices: a unified command header (UCMD) with fields including: media type flag, physical queue depth, wake-up latency, and unified LBA base address; and exposes the address of the newly added device upwards. When an upper-layer application initiates a block device read / write request, the protocol conversion module of the polling management layer intercepts the request. Based on the target logical block address (LBA) of the request, it queries the internally maintained "unified address mapping table" to determine the physical device type and its physical address corresponding to the LBA. Subsequently, the protocol conversion module encapsulates the parameters of the original request into a unified command header (UCMD), where the "media type flag" field indicates that the target device is an SSD, and puts the UCMD into the request queue of the corresponding SSD device for processing by the newly added polling management layer.

[0216] 2.2 After implementing the flexible management capability of the underlying hardware resource pool, the next step is to logically divide the high-frequency storage pool, mid-frequency storage pool, and low-frequency storage pool, that is, to logically divide the allocation of the number of SSDs, HDDs, and Blu-ray storage devices in the resource pool:

[0217] Below is an example of the logical partitioning of high-frequency storage pool, mid-frequency storage pool, and low-frequency storage pool:

[0218] We will use the following business data as an example to illustrate our specific scheme for allocating high-frequency storage pools, mid-frequency storage pools, and low-frequency storage pools.

[0219] Assuming our business data consists of 2TB of hot data, 2TB of warm data, and 6TB of cold data, we'll implement 100% redundancy for the business system. This means hot data will be stored in a high-frequency storage pool (requiring 4TB of storage), warm data in a mid-frequency storage pool (requiring 4TB of storage), and cold data in a low-frequency storage pool (requiring 12TB of storage). This redundancy will be used for new data additions to the business system and for data flow resulting from changes in hot, warm, and cold data tags. The requirements are shown in the table below.

[0220]

[0221] Suppose we have 5 magneto-optical integrated storage devices, each with three types of storage media, and the configuration of each device is as follows:

[0222] NVMe SSD: 1TB (High Performance)

[0223] SATA HDD: 2TB (Large Capacity)

[0224] Blu-ray disc library: 4TB (low-cost archiving)

[0225] Additionally, we have a separate business server A hosting the system's backend services, accessing five storage devices to retrieve data. Since these five storage devices may be located in different data centers or server rooms, the network latency from business server A to the five storage devices D1, D2, D3, D4, and D5 is not uniform, let's assume they are respectively... =1ms =1.2ms =1.3ms =1.5ms =1.7ms.

[0226] (1) High-frequency storage pool construction method

[0227] The design principle of high-frequency memory pools is:

[0228] 1) SSDs are the primary storage source, with HDDs as a secondary source. The storage capacity of SSDs should be greater than that of HDDs. In this example, the high-frequency storage pool requires a total of 4TB of storage devices, so we designed 3TB of SSDs and 1TB of HDDs.

[0229] 2) From all available devices, select the one with the shortest access latency for business server A. Here we need a 3TB SSD and a 1TB HDD; prioritize the device with the lowest latency, D1 (latency). D2 (delay) D3 SSD and D1 (latency) Of the 2TB HDDs, 1TB was chosen (because...) < < < < These devices are used to construct high-frequency storage pools. See the example highlighted in bold below:

[0230]

[0231] (2) Construction method of intermediate frequency memory pool

[0232] The design principle of the intermediate frequency layer is:

[0233] 1) After the high-frequency storage pool is selected, the mid-frequency storage pool will be selected from the remaining SSD, HDD, and Blu-ray storage.

[0234] 2) The equipment in the intermediate frequency storage pool is mainly HDD, and is a mixture of SSD, HDD and Blu-ray media in a certain proportion. HDD storage devices are required to be the most numerous. Here, it is designed with 1T SSD, 2T HHD and 1T Blu-ray.

[0235] 3) From the remaining devices, select the device with the shortest access latency for business server A. The yellow highlighted areas represent the high-frequency resource pool that has already been used. We will use the 1TB SSD from D4 (because...). < The 2T HHD is taken from the remaining 1T in D1 and 1T in D2, and the Blu-ray is taken from 4T in D1, taking 1T. Select the parts marked in bold below.

[0236]

[0237] (3) Construction method of low-frequency storage pool

[0238] The design principle of low-frequency storage pools is:

[0239] 1) After selecting the storage medium with the lowest latency in the high-frequency and mid-frequency layers, the mid-frequency storage pool will be selected from the remaining SSDs, HDDs, and Blu-ray storage.

[0240] 2) It mainly consists of Blu-ray media and HDD, with a small number of SSDs. The design includes a 1TB SSD, a 5TB HDD, and a 6TB Blu-ray disc for the low-frequency resource pool.

[0241] 3) From the remaining devices, select the device with the shortest access latency to business server A. Select the part highlighted in green below.

[0242]

[0243] Complete the physical storage resource allocation for the logical resource pool using the method described above. Store the system's hot, warm, and cold data into the corresponding logical resource pools.

[0244] 2.3 Based on data storage needs, the allocation of SSD, HDD, and Blu-ray storage in the resource pool is dynamically adjusted in real time:

[0245] Furthermore, during data storage, the large volume of data to be computed and stored means that frequent data transfers would consume significant energy. Existing technologies typically involve "data finding storage devices," meaning data is repeatedly transported, transferred, and allocated to appropriate storage devices. This data transport also consumes energy. However, this patent innovatively addresses energy consumption during data storage by having storage devices actively seek out data. By placing SSDs, HDDs, and Blu-ray discs in a single resource pool and managing data processing under a single CPU, data transfer across CPUs is eliminated. This reduces storage efficiency and energy consumption. High-frequency and mid-frequency SSDs and HDDs in the pool are automatically added to the resource pool, enabling storage devices to actively seek out data. The specific implementation is as follows:

[0246] 1. Dynamically adjust different high, medium, and low frequency pools: Evaluate the computability and editability of the stored data in the high, medium, and low frequency pools, thereby dynamically adjusting the number of SSDs and HDDs in the high and medium frequency pools. This enables storage resources to actively seek out data, without the need for time-consuming and energy-intensive data transfer.

[0247] 2. A separate global backup resource pool is established: consisting of all SSDs, HDDs, and Blu-ray devices that have been discovered but not yet allocated to any logical pool. This pool serves as a shared, elastic resource reserve.

[0248] 3. Obtain the computability and editability metrics of the data to be stored in the current data storage pool; based on the metrics, classify the data into high computability / editability, medium computability / editability, and low computability / editability.

[0249] Example of computability determination rules:

[0250] For text or structured streams, if the entropy value is < 6.0 bit / byte and ≥ 50% of keywords (JSON / XML / CSV / SQL syntax) can be parsed, it is considered high computational performance; for images / audio, if there is an editable header (PSD, XCF, WAV, MIDI) or the inter-frame compression ratio is < 60%, it is considered medium computational performance; for video streams or encrypted containers, if the intra-frame compression ratio is ≥ 60%, it is considered low computational performance.

[0251] 4. The aforementioned "newly added polling management layer" dynamically adjusts the ratio of different storage media in the current storage resource pool based on the proportional distribution of various types of data in the current storage resource pool. If step 3 determines that a large amount of highly computable text data has been injected during a certain period, the system will automatically expand the number of SSDs and HDDs in the warm pool to reduce the proportion of Blu-ray. If the injected data is low-computability video stream, the system will increase Blu-ray nodes and may even downgrade some warm pools to cold pools.

[0252] For example, for video files, the system splits them into: if the file to be stored is a video stream (low computability) → the proportion of Blu-ray in the pool increases;

[0253] If the text to be stored is subtitle text (high computability) → the proportion of SSD in the pool increases;

[0254] If the image to be stored is a cover image (computability in the middle) → the proportion of HDD in the warm pool increases; the system adjusts the resource pool allocation in reverse according to the logical object-level tags to achieve object-resource collaborative optimization.

[0255]

[0256] 5. The aforementioned "newly added polling management layer" updates the metadata: the address space of the newly added SSD in step 4 is incorporated into the unified address space of the target resource pool. The polling management layer begins to include the device in the time-sharding scheduling loop disclosed in step two of this patent. The entire process does not involve any physical movement of user data. After the new storage device SSD is mounted, these newly added SSDs can immediately serve as part of the target resource pool, carrying new write I / O requests for the pool, or serving existing highly active data in the pool through caching, data block migration, and other technologies, thereby achieving an immediate performance improvement.

[0257] III. Data Allocation and Migration Strategies

[0258] Step one involves segmenting the data and determining its temperature (warm, cold, hot) type. Step two innovates the construction of the storage resource pool, creating a robust physical environment for data storage. Step three requires designing a data allocation and migration scheme, responsible for the initial allocation of data physical blocks and how to dynamically migrate data physical blocks between logical resource pools after temperature label changes.

[0259] Current technical solutions assign temperature tags to data and then directly allocate it to the corresponding storage devices; for example, hot data is stored on SSDs, warm data on HDDs, and cold data on Blu-ray discs. This storage solution has problems. For instance, some data, although warm or cold, may be accessed frequently at certain moments, leading to significant access latency during critical times. (The problem of warm and cold data requiring instantaneous high-frequency access)

[0260] Our solution incorporates a logical storage pool concept. First, physical data blocks are mapped to high-frequency, mid-frequency, and low-frequency logical storage pools based on their temperature tags (hot, warm, cold). Second, each logical storage pool contains different storage devices; for example, the mid- and low-frequency storage pools include SSDs, HDDs, and Blu-ray discs. Based on their characteristics, physical data blocks are allocated to different storage media using a graph algorithm, addressing the issue of instantaneous high-frequency access to both warm and cold data. For instance, some data, even if it's considered cold, can be stored on SSDs or HDDs in the low-frequency storage pool. When the business system requires high-frequency access, the storage devices can still support it, significantly reducing business latency.

[0261] In addition, load balancing of storage devices is implemented within the resource pool to achieve a uniform distribution of data among different storage devices in the logical resource pool. For example, the mid-frequency storage layer includes SSDs, HDDs, and Blu-ray drives. If only low-latency data services are pursued, all data will be concentrated on SSDs, which will overload the SSDs. This will cause some data that needs to be stored on SSDs to fail to be stored because the SSDs will quickly fill up. In addition, full-load operation of SSDs will increase the overall energy consumption of the system.

[0262] The specific steps of this plan are as follows:

[0263] 1. Initial allocation strategy:

[0264] When a new data object is created, the system calculates its initial CDT score based on the first step of the CDT model. The CDT score is then compared with a threshold T. hot T warm The comparison results determine whether to write it directly into the high-frequency, medium-frequency, or low-frequency storage pool.

[0265] Example: Initial allocation of new data objects.

[0266] Suppose a new video file is added to the online education platform of a university, with the file ID Video_1001. The first step of logical segmentation yields 30 logical objects K1 and 200 physical blocks M1.

[0267] (1) CDT Initial Calculation of Initial Score:

[0268] 50 blocks CDT = 0.85 (> T) hot =0.7) → Hot data

[0269] 100 blocks CDT = 0.45 (T) warm =0.3≤CDT≤0.7)→ Temperature data

[0270] 50 blocks CDT = 0.15 (< T) warm =0.3) → Cold data

[0271] (2) Select logical storage pool:

[0272] Hot data physical block → Select high-frequency storage pool

[0273] Temperature data physical block → Select intermediate frequency storage pool

[0274] Cold data physical blocks → Select low-frequency storage pool

[0275] (3) Physical blocks are allocated storage devices within the logical storage pool.

[0276] In the previous step, physical data blocks were allocated to specific logical storage pools based on hot, warm, and cold temperature labels. After arriving at the logical storage pool, how to distribute these physical data blocks across different storage devices within the resource pool to ensure they provide the lowest possible latency data services to upper-layer applications without allocating them to high-performance storage devices? This patent proposes a graph-based physical block allocation scheme. While ensuring the quality of service for upper-layer applications (such as response latency and data availability), this scheme allows physical data blocks to be distributed as evenly as possible across SSDs, HDDs, and Blu-ray devices based on their characteristics, preventing them from clustering on SSDs and thus reducing data storage energy consumption.

[0277] Here, we will use the above-mentioned intermediate frequency storage pool as an example to illustrate the method. First, we have data physical blocks (chunks) that have been divided into physical blocks and need to be stored in the intermediate frequency storage pool. Then, our intermediate frequency storage pool has 4 storage devices:

[0278] D4 / SSD (1TB) --- Device D4's SSD, 1TB in size

[0279] D1 / HDD (1T) --- Device D1 is an HDD with a size of 1TB.

[0280] D2 / HDD (1T) --- A D2 HDD with a size of 1TB.

[0281] D1 / Blu-ray (1T) --- The Blu-ray output of device D1, 1T in size.

[0282] This process is illustrated using the method of storing 6 physical blocks (chunks) of video data as an example.

[0283] 1) Construct a graph based on physical blocks and their relationships.

[0284] First, we label these 6 chunks as V = {v1, v2, v3, v4, v5, v6}. Each chunk is defined as a node, resulting in a total of 6 nodes. We then construct the connecting edges between these nodes using two different relationships, following these rules:

[0285] Based on temporal relationships: Assuming that v1 and v2 are originally connected in a video, that is, before they are cut, they are two consecutive segments of a video, then we define that there is an edge between v1 and v2.

[0286] b. Based on semantic relationships: Calculate the semantic similarity between any two chunks according to the semantic embeddings in the metadata index of each chunk from step one:

[0287] S i,j =cos(Emb(v i),Emb(v j ))

[0288] Emb(v i ) represents the physical block of data v i semantic embedding, S i,j Indicates v i With v j Semantic similarity.

[0289] If semantic similarity S i,j If it is greater than the threshold β, then in v i With v j There is an edge between them.

[0290] Here we assume the 6 chunks are 6 segments cut from a continuous video. According to rule a, v1, v2, v3, v4, v5, and v6 should be connected sequentially by an edge. Based on this rule, the edges between our 6 nodes are as follows: Figure 3 As shown.

[0291] Calculate the semantic similarity between any two nodes according to rule b. If the similarity is greater than the threshold, there is an edge. Assume the threshold β = 0.7. The semantic similarity calculation results between each node are shown in the table below:

[0292]

[0293] Based on the semantic similarity results in the table above, we can determine which two nodes should be connected by an edge. We can see that the semantic similarity between v1 and v3, v2 and v5, and v3 and v6 is greater than the threshold of 0.7, indicating a possible connection. Since we are building an undirected graph, the edge from v1 to v3 and the edge from v3 to v1 are the same.

[0294] The final result is an undirected graph containing 6 nodes and 8 edges, as shown below. Figure 4 As shown.

[0295] 2) Node feature learning based on graph algorithms

[0296] Taking the video chunk above as an example, we believe that the features of a chunk should be related to chunks that have a temporal relationship with it, and also to chunks that are semantically similar to it. Therefore, we construct the graph according to the method above. After completing the graph construction, we define the feature vector of the current chunk based on the node attributes of the chunk itself and by aggregating the node attributes of its neighboring chunks in the graph. The specific method is as follows:

[0297] a) First, we define a chunk v using a vector. i Node attribute P(v)i ):

[0298] P(v i =[Emb(vi), F score (vi), M score (vi), B score (vi), f((vi)), size((vi))]

[0299] Where Emb(vi) is its semantic embedding, F score It is the access frequency score defined in the first step, M score It is the data modality heat score defined in the first step, B score Here, is the business characteristic score defined in the first step, f is the access frequency of the chunk, and size is the physical size of the chunk. P(vi) represents concatenating these values ​​to form a vector representing the node attributes of chunk vi. For example, for node v2, its own node attribute vector P(v2) = [Emb(v2), F score (v2), M score (v2), B score (v2), f((v2)), size((v2))].

[0300] Emb(v2) is a 256-dimensional semantic embedding, followed by... The five numbers, access frequency and size, together form a 231-dimensional vector to represent the node attributes of node v2.

[0301] b) Find the first-order neighbor chunk of the current chunk in the graph.

[0302] First-order neighbors are those that are directly connected to the current node. For example, the first-order neighbors of node v2 are v1, v3, and v5.

[0303] c) Aggregate the node attribute vectors of neighboring nodes and combine them with the node attribute vector of the current node to construct the feature vector of the current node.

[0304] The calculation method is as follows:

[0305] First, consider the node attribute vector P(v) of all first-order neighbor nodes. i Taking node v2 as an example, its first-order neighbors are v1, v3, and v5. Their node attribute vectors are P(v1), P(v3), and P(v5). The average of these vectors is then taken.

[0306] P mean =MEAN(P(v1), P(v3), P(v5))

[0307] Current node feature vector

[0308]

[0309] in This represents the node attribute vector of the current node. The mean of the node attribute vectors of first-order neighbor nodes Perform a left and right join operation to transform it into a vector of 2^31 + 2^31 = 462 dimensions. These are trainable weights, and ReLU is the activation function. After the above operations, we obtain a feature vector for the current node.

[0310] d) Construct a node classification model

[0311] This step trains a classifier to predict which physical storage device each video physical block chunk should be allocated to.

[0312] In the output of the previous step On top of this, add a classification header (fully connected layer):

[0313]

[0314] Indicates the predicted value. and These are the parameters that need to be trained.

[0315] Training: The model parameters are trained using a labeled dataset (based on historical data, determining which storage device is best suited for storing video physical blocks, with manually labeled categories). and .

[0316] e) Physical block allocation storage device prediction

[0317] Prediction Phase: For a batch of video physical blocks, a flowchart is constructed based on the above process, node feature vectors are calculated, and a classification model is used to predict the appropriate storage device for the current physical block. Here, we still take the mid-frequency storage pool mentioned above as an example, including D4 / SSD, D1 / HDD, D2 / HDD, and D1 / Blu-ray. After model prediction, the results are obtained, for example:

[0318]

[0319] (4) Metadata index update and storage

[0320] Once the storage location of a physical block is determined, its metadata is updated and written to the SSD device in its logical storage pool.

[0321] Example:

[0322] Update the address where a physical block metadata index is stored, as shown below:

[0323] Node V3 corresponds to oid = "Video_1001 / logic_0009 / chunk_0003"

[0324] metadata[oid] = [32MB, "warm", "D4 / SSD", [semantic vector], "0x5b2e..."]

[0325] This indicates that the physical block is the 0003rd physical block of logical object 0009 of Video_1001, and it is warm data with a size of 32M. D4 / SSD: indicates that it is stored on the SSD of physical device 4.

[0326] Finally, the metadata index of this physical block is written to the SSD in the intermediate frequency storage pool for subsequent lookups of that physical block. (Metadata for each logical storage pool is uniformly stored on the SSD device at this layer.)

[0327] 2. Dynamic migration strategy:

[0328] The system's backend service will periodically (or under specific event triggers) recalculate the CDT score of existing data objects.

[0329] Migration triggering mechanism:

[0330] (1) Upgrade migration (cold to warm, warm to hot): When a data object located in a lower layer has its CDT (Data Content Time) exceed the upper limit threshold of its layer, the system background service will immediately trigger an upgrade migration to move it to a higher performance layer to meet the needs of subsequent continuous access. The data can be accessed directly after migration without the need for re-migration.

[0331] Upgrade and migration example:

[0332] Assume T warm =0.3, T hot =0.7,

[0333] The cold data block "Video_0008 / logic_0002 / chunk_0003" (original CDT=0.2) was suddenly accessed at high frequency. The migration steps are as follows:

[0334] a) Access triggers CDT recalculation: Real-time access causes CDT to spike to 0.75 (> T). warm =0.3)

[0335] b) Immediately trigger upgrade migration: IDME detects that CDT > cold data layer threshold limit and adds the block to the upgrade migration queue.

[0336] c) The migration engine copies data from the low-frequency pool (SATA HDD) to the mid-frequency pool (SAS HDD).

[0337] d) Based on the graph algorithm described above, its suitable physical storage location is predicted to be D2 / HDD.

[0338] e) Seamless switching of metadata and access: Update physical location: "D4 / HDD" → "D2 / HDD", and subsequent accesses will directly point to the new location without migration delay.

[0339] f) Resource release: The original cold data location space is marked as reclaimable.

[0340] (2) Degradation migration (hot to warm, warm to cold): When the CDT score of a data object remains below the lower limit threshold of its layer for a period of time (for example, the CDT of hot data is lower than T for 3 consecutive periods), hot The system's backend service will add it to the downgrade migration queue.

[0341] Example: Hot data block Video_0010 / logic_0005 / chunk_0007 (original CDT=0.8) has not been accessed for 3 consecutive cycles.

[0342] a) Periodic CDT reassessment:

[0343] Period 1: CDT = 0.65 (< T) hot =0.7) → Mark "Observation Status"

[0344] Period 2: CDT=0.58 → Maintain observation

[0345] Period 3: CDT=0.52 → Triggering downgrade conditions

[0346] b) Add to the downgrade migration queue

[0347] c) The system background service moves it out of the high-frequency pool (D1 / SSD), predicts its suitable physical storage location in the mid-frequency pool (D4 / SSD) using the graph algorithm mentioned above, and then migrates it to that location.

[0348] d) Metadata Update: Metadata update physical location: "D1 / SSD" → "D4 / SSD"

[0349] e) Resource optimization: Free up 40MB of space in the high-frequency pool for use with new thermal data.

[0350] 3. Logical storage pool capacity monitoring and dynamic expansion

[0351] The system backend monitors the usage of the high-frequency, mid-frequency, and low-frequency storage pools. If the usage of any logical storage pool exceeds 90%, the logical storage pools are re-partitioned according to the method in step two.

[0352] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0353] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0354] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0355] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0356] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A logical layering and data scheduling method for a magneto-optical integrated storage device, characterized in that, include: Multimodal files are semantically split into multiple logical objects, and each logical object is then divided into multiple physical blocks; The comprehensive data temperature value of each physical block is calculated based on multidimensional features, and the physical block is divided into hot data, warm data or cold data according to the comprehensive data temperature value, which serves as its temperature label. Build a data index for each physical block that includes access ID, block size, block location, temperature label, and semantic embedding; The various heterogeneous storage media in the magneto-optical integrated storage device are combined according to a preset storage media ratio to construct a logical storage pool that includes at least a high-frequency storage pool, a medium-frequency storage pool, and a low-frequency storage pool; the logical storage pool is uniformly managed by the same central processing unit through a polling management layer; Based on the temperature tag of the physical block, the physical block is initially allocated to the corresponding high-frequency, medium-frequency or low-frequency storage pool, and the physical blocks in each storage pool are allocated to the specific storage medium contained in that storage pool; the comprehensive data temperature value of each physical block is periodically recalculated, and the physical blocks are dynamically migrated between different storage pools according to the change of the comprehensive data temperature value; The multidimensional features include at least access frequency features based on physical block access history, data modality heat features based on the file modality to which the physical block belongs and the access heat of that file modality, and business characteristic features based on physical block service business; by combining access frequency features, data modality heat features, and business characteristic features, a comprehensive data temperature value for each physical block is calculated; Then, the combined data temperature value is compared with the preset temperature threshold to divide the physical block into hot data, warm data, or cold data; The method for calculating the comprehensive data temperature value is as follows: in, For the comprehensive data temperature value, For access frequency score, The data modality heat score, Assigning scores based on business characteristics. , , These are the weighting factors for the access frequency score, data modality popularity score, and business characteristic score, respectively. ; The access frequency score is calculated based on the access history of the physical block by introducing a time decay effect. The calculation formula is as follows: in, The current access time. The last access time, This represents the current access frequency score for the physical block. This is the access frequency score calculated based on the last access to the physical block. The time interval between the current access and the previous access. Here, A is the time decay coefficient, and A is the gain in popularity per visit. The data modality heat score is calculated based on the file modality to which the physical block belongs and the access heat of that file modality. The calculation formula is as follows: in, This represents the current data modality heat score for the physical block. This is the modal heat score of the data calculated during the last access to the physical block. Modal influence factor, The rate of increase in the popularity score. for The number of times the file mode to which the physical block belongs is accessed within the time interval. The rate at which the heat score decreases; The business characteristic score is calculated based on a weighted average of data source, service level agreement, and regulatory compliance, using the following formula: in, Score based on data source. For service level agreement scores, The score represents regulatory compliance, and N(.) is the normalization function. , , These are the weighting coefficients.

2. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 1, characterized in that, The step of splitting the multimodal file into multiple logical objects according to semantics, and then further dividing each logical object into multiple physical blocks, specifically includes: The multimodal files include video files, audio files, and document files. For video files, a scene detection algorithm based on content similarity is used for segmentation, identifying scene boundaries in the video and dividing the video into multiple video logical objects based on these boundaries. For audio files, a speech activity detection algorithm is used for segmentation, dividing the audio into multiple audio logical objects based on silence segment boundaries. For document files, the chapter or page structure of different document formats is analyzed, and the document is divided into multiple document logical objects based on the chapter or page structure. Set a fixed segmentation threshold. For logical objects whose size exceeds the segmentation threshold, they are segmented into multiple physical blocks according to the segmentation threshold. The segmentation thresholds for video logical objects, audio logical objects, and document logical objects are set independently.

3. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 1, characterized in that, Two dynamic thresholds T are preset hot and T warm , the physical block is divided according to data temperature in the following way: If CDT > T hot Then the physical block is divided into hot data; If T warm < CDT ≤ T hot Then the physical block will be divided into temperature data; If CDT ≤ T warm Then the physical block will be divided into cold data.

4. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 1, characterized in that, The structure of the data index is as follows: {OID:[size,temperature,physicallocation,semantic embedding,checksum]} Wherein, OID represents the access ID of the physical block, OID = original file ID + logical object number + physical block sequence number, size represents the size of the physical block, temperature represents the temperature label of the physical block, physical location represents the physical storage location of the physical block, and semantic embedding represents the semantic embedding vector of the physical block.

5. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 1, characterized in that, The heterogeneous storage media include at least three different storage media types: solid-state drives (SSDs), hard disk drives (HDDs), and Blu-ray discs. The magneto-optical integrated storage device combines SSD, HDD, and Blu-ray disc storage media of three types, and constructs a high-frequency storage pool, a mid-frequency storage pool, and a low-frequency storage pool according to a preset storage media ratio based on the network latency of accessing each storage device. Among them, the high-frequency storage pool uses SSD as the main storage medium and configures a small number of HDDs according to a set ratio; the mid-frequency storage pool uses HDD as the main storage medium and configures some SSDs and Blu-ray discs according to a set ratio; the low-frequency storage pool uses Blu-ray discs and HDDs as the main storage media and configures a small number of SSDs according to a set ratio.

6. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 5, characterized in that, The method for constructing the high-frequency storage pool is as follows: prioritize the storage device with the shortest access latency, use SSD as the main storage medium and HDD as the auxiliary storage device, and the capacity ratio of SSD is greater than that of HDD. The method for constructing the intermediate frequency storage pool is as follows: after selecting the high frequency storage pool, the storage device with the shortest access latency is selected from the remaining storage devices, with HDD as the main storage medium, and SSD and Blu-ray discs are mixed and configured, with HDD having the largest capacity. The method for constructing the low-frequency storage pool is as follows: after selecting the high-frequency storage pool and the medium-frequency storage pool, the storage medium with the shortest access latency is selected from the remaining storage media, with Blu-ray discs and HDDs as the main storage media, and a small number of SSDs configured, with the capacity of Blu-ray discs accounting for a larger proportion than that of HDDs and the capacity of SSDs accounting for the smallest proportion.

7. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 5, characterized in that, The polling management layer is implemented in the form of a kernel-mode driver and adopts a multi-threaded architecture, including a high-priority real-time thread for polling solid-state drives, a worker thread for processing mechanical hard drive requests, and a background management thread for managing the status of Blu-ray discs. The polling management layer performs the following operations: A time-slicing mechanism is used to assign different time-slice weights to storage devices with different storage media types; Based on time slice weights, the status of each storage device is periodically polled, including ready, busy, and sleeping. Only schedule input / output requests for storage devices in the ready state, and send wake-up commands to storage devices in the hibernation state and suspend the corresponding requests until their state becomes ready; The polling management layer is also used to: intercept and parse physical block read / write requests initiated by upper-layer applications, convert them into protocol commands corresponding to the target storage device type, and distribute them to the corresponding storage device queue for processing.

8. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 1, characterized in that, A graph algorithm is used to allocate physical blocks within each storage pool to the specific storage media contained in that storage pool, specifically including: A graph structure is constructed based on the temporal relationship and semantic similarity of physical blocks, wherein there are edges between temporally continuous physical blocks or physical blocks with semantic similarity greater than a preset threshold; Feature learning is performed on each physical block node in the graph, and the node's own attributes and the attributes of its first-order neighbor nodes are aggregated to generate a feature vector. The node attributes include semantic embedding vector, access frequency score, data modality heat score, business characteristic score, access frequency and physical size. Based on feature vectors, a classification model is used to predict the target storage medium of physical blocks, enabling the allocation of physical blocks on SSDs, HDDs, and Blu-ray discs within the storage pool.

9. The logical layering and data scheduling method for the magneto-optical integrated storage device according to claim 1, characterized in that, When the overall data temperature value of a physical block exceeds the upper limit threshold of its storage pool, an upgrade migration is triggered: the physical block is immediately migrated from the current storage pool to a higher-performance storage pool. The migration process is executed asynchronously in the background. After the migration is completed, the data index of the physical block is updated to point to the new storage location. When the overall data temperature value of a physical block remains below the lower limit threshold of its storage pool for a preset period, a degradation migration is triggered: the physical block is migrated from the current storage pool to a lower-performance storage pool, and the original storage location space is released and the data index is updated after the migration is completed. The system monitors the percentage of used capacity in high-frequency, mid-frequency, and low-frequency storage pools in real time. When the percentage of used capacity in any storage pool exceeds a preset threshold, the storage media are recombined and the storage pools are reconstructed.

Images