Distributed storage resource intelligent scheduling method and device

Abstract

The invention provides a distributed storage resource intelligent scheduling method and device, and relates to the technical field of data processing, and the method comprises the steps: carrying out the feature splicing of feature vectors of different modes, so as to obtain a multi-mode feature vector; predicting the user satisfaction based on the multi-modal feature vector and the resource adjustment parameter; establishing a resource demand priority mapping table under different service scenes according to the user satisfaction; constructing a multi-objective optimization model according to the resource demand priority mapping table; on the basis of the multi-objective optimization model, predicting the resource state and the load condition of each node in the distributed storage system; and according to the resource state and the load condition of each node, lightweight rule scheduling is carried out to obtain a preliminary scheduling scheme. According to the invention, intelligent scheduling management of distributed storage resources is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for intelligent scheduling of distributed storage resources. Background Technology

[0002] As the infrastructure for modern cloud computing and big data processing, distributed storage systems have evolved from early open-source solutions such as HDFS and Ceph to enterprise-level solutions that support multi-protocol integration, high performance, high availability, and fault tolerance.

[0003] However, with the increasing demand for training and inference of large-scale artificial intelligence models, continuous technological iteration, and the emergence of diverse business scenarios such as databases and office applications, traditional distributed storage systems face multiple challenges: Low resource utilization: In AI large-scale model training scenarios, GPU utilization is typically below 30%, resulting in significant resource waste. Poor small file processing efficiency: Large model training datasets often contain billions to tens of billions of small files, and traditional distributed systems are inefficient at reading and writing small files. Insufficient intelligent resource scheduling: Existing scheduling strategies are mostly based on fixed rules or simple predictions, lacking a deep understanding and dynamic adaptability to complex workloads. For example, OLTP databases require low-latency, high-IOPS SSD storage, while video conferencing systems require high-bandwidth, large-capacity object storage.

[0004] Traditional scheduling methods cannot intelligently adjust resource allocation strategies according to dynamic changes in business scenarios, resulting in resource waste and performance bottlenecks. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for intelligent scheduling of distributed storage resources, thereby realizing intelligent scheduling and management of distributed storage resources.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a method for intelligent scheduling of distributed storage resources, the method comprising: Collect multimodal data from a distributed storage system; analyze the multimodal data to extract multimodal data features, including text features and time-series features, to obtain feature vectors for different modalities; construct a multidimensional data space based on the storage server rack location and physical partitions of the storage devices, and construct a tree-like abstract data structure based on this multidimensional data space; divide the abstract data structure into multiple partition units with physical server nodes as the basic unit, divide the partition units according to the data distribution characteristics to form multiple sub-regions, map the data to the corresponding sub-regions, and obtain resource adjustment parameters based on the data characteristics of each sub-region; Feature vectors from different modalities are concatenated to obtain multimodal feature vectors; Based on multimodal feature vectors and resource adjustment parameters, predict user satisfaction; based on user satisfaction, establish a resource demand priority mapping table for different business scenarios; Construct a multi-objective optimization model based on the resource demand priority mapping table; Based on a multi-objective optimization model, the resource status and load of each node in a distributed storage system are predicted. Based on the resource status and load of each node, lightweight rule scheduling is performed to obtain a preliminary scheduling scheme.

[0007] Secondly, a distributed storage resource intelligent scheduling device includes: The acquisition module is used to acquire multimodal data from the distributed storage system; analyze the multimodal data to extract multimodal data features, including text features and time-series features, to obtain feature vectors for different modalities; construct a multidimensional data space based on the storage server rack location and storage device physical partitions, and construct a tree-like abstract data structure based on this multidimensional data space; divide the abstract data structure into multiple partition units with physical server nodes as the basic unit, divide the partition units according to the data distribution characteristics to form multiple sub-regions, map the data to the corresponding sub-regions, and obtain resource adjustment parameters based on the data characteristics of each sub-region; The concatenation module is used to concatenate feature vectors from different modalities to obtain multimodal feature vectors. The module is used to predict user satisfaction based on multimodal feature vectors and resource adjustment parameters; to establish a resource demand priority mapping table for different business scenarios based on user satisfaction; and to build a multi-objective optimization model based on the resource demand priority mapping table. The prediction module is used to predict the resource status and load of each node in the distributed storage system based on a multi-objective optimization model; and to perform lightweight rule scheduling based on the resource status and load of each node to obtain a preliminary scheduling scheme.

[0008] The above-described solution of the present invention has at least the following beneficial effects: By combining large-scale models with distributed storage technology, efficient management and use of distributed storage system resources can be achieved, improving the performance, stability, and ease of use of the storage system, and enabling predictive maintenance. Through historical data and real-time usage patterns of large-scale models, the risk of equipment failure can be predicted, reducing unplanned downtime losses. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the intelligent scheduling method for distributed storage resources provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a distributed storage resource intelligent scheduling device provided in an embodiment of the present invention. Detailed Implementation

[0010] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0011] like Figure 1 As shown, embodiments of the present invention propose an intelligent scheduling method for distributed storage resources, the method comprising the following steps: Step 1: Collect multimodal data from the distributed storage system; analyze the multimodal data to extract multimodal data features, including text features and time-series features, to obtain feature vectors for different modalities; construct a multidimensional data space based on the storage server rack location and storage device physical partitions, and construct a tree-like abstract data structure based on this multidimensional data space; divide the abstract data structure into multiple partition units with physical server nodes as the basic unit, divide the partition units according to the data distribution characteristics to form multiple sub-regions, map the data to the corresponding sub-regions, and obtain resource adjustment parameters based on the data characteristics of each sub-region; Step 2: Concatenate the feature vectors of different modalities to obtain a multimodal feature vector; Step 3: Based on multimodal feature vectors and resource adjustment parameters, predict user satisfaction; based on user satisfaction, establish a resource demand priority mapping table for different business scenarios. Step 4: Construct a multi-objective optimization model based on the resource demand priority mapping table; Step 5: Based on the multi-objective optimization model, predict the resource status and load of each node in the distributed storage system; Step 6: Perform lightweight rule scheduling based on the resource status and load of each node to obtain a preliminary scheduling scheme.

[0012] In this embodiment of the invention, a multi-dimensional data space is constructed based on the location of the storage server rack and the physical partitioning of the storage device. A tree-like abstract data structure is used to divide the partitioned units and sub-regions, allowing data mapping to deeply match the physical storage architecture. This avoids the data location confusion caused by traditional disordered storage, enabling rapid locking of the physical node where the data resides during subsequent resource scheduling, reducing the time loss of cross-node data lookups, and improving the basic efficiency of data access and scheduling. For multi-modal data, textual time-series features are extracted, and a unified multi-modal feature vector is formed through feature concatenation. Compared to traditional single-data-type analysis, this can more completely capture the data attributes in the storage system, such as textual business requirements describing time-series I / O load changes, reducing scheduling decision bias caused by missing data dimensions. Based on the multi-modal feature vector and resource adjustment parameters, user satisfaction is predicted, thereby establishing a resource requirement priority mapping table for different business scenarios. This shifts priority allocation from fixed rule-driven to user experience and business characteristic-driven. For example, in database scenarios, due to users' high satisfaction with low latency requirements and their sensitivity to latency, high resource priority can be automatically assigned, avoiding the problem of insufficient core business resources caused by the one-size-fits-all priority setting in traditional scheduling.

[0013] In this embodiment, a multi-objective optimization model is constructed by combining a resource demand priority mapping table. This model can simultaneously consider multiple scheduling objectives, such as high resource utilization and low latency service priority matching, rather than the traditional single-minded pursuit of load balancing. Based on the multi-objective optimization model, the resource status and load of each node are predicted, which can identify node overload risks in advance. For example, if a node's CPU utilization is about to surge or its resources are idle, or if a node's memory usage is low for a long period of time, compared to traditional post-event scheduling that adjusts only after a node is overloaded, resource allocation can be planned in advance to avoid node crashes due to sudden load, reduce the probability of system failure, and enhance the stability of the storage system. A preliminary scheduling scheme is generated through lightweight rule scheduling, which can quickly respond to resource demands without complex calculations. This is suitable for scheduling scenarios with high-frequency, low-latency demands in storage systems, such as short-term burst I / O requests. The preliminary scheduling scheme can quickly balance the basic load of each node.

[0014] In this embodiment, user satisfaction is used as the core basis for resource priority allocation, so that the scheduling strategy directly serves to improve user experience. For example, for services that are sensitive to user operation response latency, such as real-time office document editing, high-speed storage layer resources can be allocated preferentially through scheduling to reduce document saving and opening latency, improve user experience, and enhance the storage system's adaptability to user needs. From data feature extraction and priority allocation to multi-objective optimization, the entire process revolves around the characteristics of different business scenarios. For example, when the system adds video conferencing services, the scheduling method can analyze the multimodal data characteristics of its high-bandwidth and large-capacity storage, automatically match the corresponding resource priorities and scheduling strategies, reduce the scheduling configuration costs when expanding services, and improve the system's adaptability to dynamic changes in services.

[0015] In a preferred embodiment of the present invention, step 1 above involves collecting multimodal data from a distributed storage system; analyzing the multimodal data to extract multimodal data features, including text features and time-series features, to obtain feature vectors for different modalities. The multimodal data includes CPU and GPU utilization metrics from computing nodes, I / O performance data from SSDs and HDDs from storage nodes, and bandwidth and latency data from network devices. Multimodal data is collected from the distributed storage system, specifically from the associated compute nodes, storage nodes, and network devices. Data from compute nodes includes CPU and GPU utilization metrics, requiring real-time recording of resource usage changes for both types of hardware during different computing tasks. Data from storage nodes covers I / O performance data for both SSDs and HDDs, including parameters such as read / write response time, read / write operations per second, and data throughput. Data from network devices includes bandwidth usage and latency data, requiring the capture of real-time bandwidth usage and data transmission / reception latency during data transmission.

[0016] The data collection process involves deploying hardware monitoring plugins on computing nodes to capture real-time changes in CPU and GPU utilization and generate logs; configuring I / O performance monitoring modules on storage nodes to continuously collect various I / O performance parameters of SSDs and HDDs; and recording bandwidth usage and data transmission latency data on network devices. All data collected from each node and device is then aggregated at a unified data collection center for standardized data format processing to ensure consistency in subsequent analysis. When analyzing and extracting features from multimodal data, text-based feature-related data, such as descriptions of computing task types and annotations of the purpose of stored data, undergo preprocessing to remove meaningless characters and standardize expression. Keyword extraction techniques are then used to identify core information such as task attributes and data categories. Semantic analysis is combined to determine implicit features such as task urgency and data importance. Finally, this text information is converted into numerical form to form text feature vectors.

[0017] For time-series data, including real-time changes in CPU and GPU utilization of computing nodes, dynamic I / O performance data of SSDs and HDDs of storage nodes, and fluctuations in network device bandwidth and latency, the continuously collected data is segmented and processed according to preset time intervals. For the data in each time period, the average, maximum, minimum, and rate of change are calculated to identify the patterns of data changes over time, such as periodic peaks in CPU utilization, sudden drops in SSDI / O performance, and continuous growth trends in bandwidth usage. At the same time, abnormal data points are captured, such as instantaneous spikes in GPU utilization and sudden increases in network latency. This time-series information is quantified into an ordered numerical sequence to form a time-series feature vector. For other auxiliary modal data, such as the execution time of computing tasks, the file format of stored data, and the transmission protocol type of network data, they are classified and encoded according to preset rules, and the associated auxiliary features are extracted and converted into corresponding numerical features to form feature vectors for other modalities.

[0018] Through the above process, text feature vectors, time-series feature vectors, and other modal feature vectors corresponding to data from computing nodes, storage nodes, and network devices are obtained respectively. Each vector accurately reflects the core attributes of multimodal data from different sources.

[0019] In a preferred embodiment of the present invention, a multi-dimensional data space is constructed using the storage server rack location and physical partitioning of storage devices as references for multi-modal data. A tree-like abstract data structure is then constructed based on this multi-dimensional data space. Within the abstract data structure, multiple partition units are divided using physical server nodes as basic units. These partition units are further divided into multiple sub-regions based on data distribution characteristics. Data is mapped to the corresponding sub-regions, and resource adjustment parameters are obtained based on the data characteristics of each sub-region, including: Step 1.1 involves preprocessing the multimodal data collected from compute nodes, storage nodes, and network devices, and organizing it according to its physical attributes and logical relationships. A multidimensional data space is constructed using the storage server rack location and storage device physical partitions as spatial references. Specifically, this includes deploying Prometheus and NodeExporter monitoring tools on each compute node to collect CPU and GPU utilization data in real time. CPU utilization is collected by core, including real-time single-core utilization, 5-minute average utilization, and peak utilization within 1 hour. GPU utilization includes compute core utilization, video memory usage, and video memory bandwidth consumption. Data collection frequency is dynamically configured based on business scenarios, with a collection interval of 1 second. For scenarios such as database OLTP and office applications, the collection interval is set to 5 seconds to ensure a balance between data timeliness in high-frequency scenarios and resource conservation in low-frequency scenarios. An I / O performance monitoring module is configured on the storage node to acquire I / O performance data for SSDs and HDDs respectively. SSD data includes random read / write IOPS, sequential read / write throughput, average read / write response time, and write amplification factor. HDD data includes sequential read / write rate, seek time, rotational latency, and cache hit rate. The collection interval is uniformly set to 2 seconds, and the physical partition identifier of the storage device corresponding to the data is recorded synchronously to provide a basis for subsequent space mapping.

[0020] Bandwidth and latency data are collected from network devices such as switches and routers. Bandwidth data includes real-time uplink bandwidth usage, downlink bandwidth usage, and bandwidth utilization for each port. Latency data includes end-to-end data transmission latency, packet loss rate, and retransmission rate. The collection interval is set to 1 second, and the hardware identifiers of the source and destination nodes of the data transmission are recorded to ensure that the data is correlated with the network topology. Invalid values ​​are filtered by combining IQR outlier detection and Z-score methods. For example, data that exceeds a reasonable range, such as CPU utilization greater than 100%, GPU memory usage negative, or SSDIOPS remaining at 0 for 30 consecutive seconds, are directly marked as invalid and removed. Duplicate data is identified by comparing the data timestamp with the source identifier (such as device number and collection time). The original data collected for the first time is retained, and subsequent duplicate records are deleted to avoid data redundancy affecting the accuracy of the analysis.

[0021] Numerical data such as CPU utilization and bandwidth usage are uniformly retained to two decimal places; timestamp data uses UTC time format and is accurate to milliseconds; device identification data is encoded according to the rules of rack number, server number, device type, and port / partition number, for example, R01-S03-SSD-P02 represents partition 2 of SSD 1 on server 3 in rack 1. Data that does not conform to the standard format is automatically corrected by a format conversion tool, such as converting local timestamps to UTC time and supplementing non-standard device identifiers with uniform encoding to ensure that the data format is consistent across different sources.

[0022] Physical attribute tags are added to each preprocessed data entry, covering the location and type of hardware devices. Hardware device location tags are generated based on the actual deployment location of the data acquisition nodes. For example, compute nodes correspond to R02-S05 (rack 2, server 5), storage nodes correspond to R03-S08-SSD01 (rack 3, server 8, SSD 1), and network devices correspond to R01-SW02-P15 (rack 1, switch 2, port 15). Device type tags are further subdivided by compute, storage, and network, such as CPU, GPU, SSD, HDD, and switch port, to determine the hardware category to which the data belongs.

[0023] Logical associations are established through business task identifiers and data dependency identifiers. The business task identifiers are the business scenario classifications mentioned in the above embodiments, such as AI training tasks being labeled as T-AI-001, office document storage tasks as T-OFF-005, and database OLTP tasks as T-DB-002. The dependencies between data are analyzed. For example, the GPU utilization data of the AI ​​training task is dependent on the SSDI / O data of the corresponding storage node. The association is established by adding a dependency identifier such as DT-AI-001-01 to determine the logical calling relationship between the data.

[0024] The storage server rack locations are divided into three levels of coordinates: rack, server, and slot. Rack numbers are sequentially numbered starting from R01. Server numbers within each rack start from S01. Slot numbers within each server are numbered vertically from P01 (top) to P10 (bottom). For example, the vertical coordinates for slot 3 on server 5 in rack 3 are R03-S05-P03. Each storage server has a unique vertical coordinate based on its actual deployment location.

[0025] Each physical partition of a storage device is numbered sequentially starting from 001, and the start address, end address, and capacity of each partition are recorded. For example, the horizontal coordinate of physical partition 3 of SSD 1 is SSD01-P003-0x00030000-0x0005FFFF-50GB, which includes device identifier, partition number, address range, and capacity information to ensure that each physical partition corresponds to a unique horizontal coordinate.

[0026] The organized multimodal data is mapped to a multidimensional space using a combination of vertical and horizontal coordinates, while simultaneously associating data types with business-related information. For example, a piece of SSDI / O performance data has a vertical coordinate of R03-S08-P04 and a horizontal coordinate of SSD01-P003-0x00030000-0x0005FFFF-50GB. The data type is labeled as SSDI / O, and the business association is labeled as T-AI-001. Ultimately, a multidimensional data space is constructed, comprising four dimensions: vertical coordinates, horizontal coordinates, data type, and business association. This enables precise positioning of data based on physical references, providing a spatial foundation for subsequent tree-like abstract data structure construction and resource scheduling.

[0027] Step 1.2: Based on the multidimensional data space, construct a tree-like abstract data structure for resource feature expression, with physical server nodes as root nodes and storage devices as leaf nodes. Specifically, this includes extracting the core association information between physical server nodes, storage devices, and multimodal data from the constructed multidimensional data space.

[0028] First, using the physical server node as the core index, the vertical coordinates (rack, server, and slot) corresponding to the server in the multidimensional data space are matched to filter out all associated storage devices (SSD, HDD) under that coordinate. Second, based on the horizontal coordinates (device, partition, address, and capacity) of the storage device, parameters such as the I / O performance data and the number of physical partitions of the corresponding storage device are associated. Finally, combined with the business association identifiers of the multimodal data (such as T-AI-001, T-DB-002), a one-to-one correspondence between physical server nodes, storage devices, and multimodal data (including business attributes) is established to form an association table, providing data support for the construction of the tree structure.

[0029] Each physical server node is designated as an independent root node in a tree-like abstract data structure. The root node is uniquely identified by a code consisting of the rack number and server number (e.g., R03-S08), ensuring consistency with the vertical coordinates of the multi-dimensional data space. The basic server information stored in the root node must cover three core categories: first, identification information, including the server number, rack location (e.g., R03), and slot number (e.g., P04); second, hardware configuration information, including the number of CPU cores, total memory capacity, and number of network ports; and third, associated device information, including the number of directly connected storage devices and the unique identifier of each storage device (e.g., SSD01, HDD05). All information is stored in key-and-value pair format for easy subsequent quick querying and retrieval.

[0030] Each storage device (SSD, HDD) is set as a leaf node in a tree-like abstract data structure. The encoding rules for the leaf nodes are server code, device type and device serial number (such as R03-S08-SSD-01), which are directly associated with the horizontal coordinates of the multidimensional data space.

[0031] The specific parameters of the devices stored in the leaf nodes need to be refined according to the device type: For SSDs, the device type (NVMeSSD / SATASSD), number of physical partitions, capacity and address range of each partition, current storage data type (such as AI training dataset, office documents), and real-time I / O performance indicators (such as random read / write IOPS, average response time) need to be recorded; for HDDs, the device type (SATAHDD / SASHDD), number of physical partitions, partition capacity and address range, current storage data type (such as cold data archive), sequential read / write speed and seek time need to be recorded; all parameters are stored in the format of device attributes, parameter values ​​and update time to ensure the timeliness and traceability of parameters.

[0032] To build a hierarchical relationship between the root node and leaf nodes, firstly, the root node is divided into first-level branches based on the physical connection method of the storage devices (such as SATA interface, PCIe interface), with each branch labeled with the interface type and interface number (such as PCIe-01, SATA-03); secondly, each first-level branch is associated with the corresponding storage device leaf node, and the interface number is matched with the physical connection identifier of the storage device to ensure that the connection relationship between the branch and the leaf node completely corresponds to the actual hardware deployment; finally, connection status identifiers (such as online / offline) and data transmission bandwidth limits are added to the hierarchical relationship to reflect the availability and performance limits of the hardware connection in real time.

[0033] This hierarchical mapping forms a three-level tree structure consisting of a root node (physical server), first-level branches (interfaces), and leaf nodes (storage devices), clearly presenting the hierarchical relationship of hardware devices in the multidimensional data space, as well as the ownership of multimodal data corresponding to each storage device.

[0034] Step 1.3: Based on the distribution density and business relevance of multimodal data in the multidimensional data space, the tree-like abstract data structure is divided into regions to form multiple partition units with physical server nodes as the basic units. Step 1.4: Perform cluster analysis on the data distribution within each partition unit, divide it into sub-regions, and map each data instance to its respective sub-region based on its physical location and access characteristics; Step 1.5: Extract the data access frequency, data size, and response time characteristics of each sub-region. Calculate the resource weight coefficient of the corresponding sub-region based on the data access frequency, data size, and response time characteristics to obtain resource adjustment parameters. Specifically, this includes: continuously collecting data access records for each sub-region, counting the total number of times various types of data in the sub-region are accessed within a preset time period, calculating the data access frequency, and recording the initiation and completion times of each access to calculate the overall data access response time of the sub-region; reading the size information of all data instances in the sub-region, summing them to calculate the data size of the sub-region, and determining the space resources occupied by data storage.

[0035] By comprehensively extracting the data access frequency, data size, and response time characteristics of sub-regions, the impact of each characteristic on resource requirements is analyzed. Sub-regions with high data access frequency have a more urgent need for storage I / O resources, sub-regions with large data sizes require more storage space resources, and sub-regions with long response times need to optimize computing or network resources to improve processing efficiency. Based on the degree of impact of each characteristic, corresponding calculation weights are assigned to data access frequency, data size, and response time. Combining the specific values ​​of each characteristic, a resource weight coefficient for each sub-region is calculated using preset evaluation rules. This resource weight coefficient is the resource adjustment parameter, which can be directly used to determine the priority of subsequent storage resource allocation and dynamically adjust the resource configuration.

[0036] In this embodiment of the invention, multimodal data is preprocessed to remove invalid values ​​and correct format differences. A multidimensional data space is constructed by combining the storage server rack location (vertical) and the physical partitioning of storage devices (horizontal). This ensures a precise correspondence between data and physical storage architecture, avoiding the disconnect between data and physical location in traditional data organization, reducing time consumption for cross-device and cross-rack data scheduling, and improving the basic efficiency of data location and scheduling. A tree structure is constructed with physical server nodes as root nodes and storage devices as leaf nodes, intuitively presenting the hierarchical relationship between hardware devices and data ownership, solving the problem of ambiguous hardware-data association in traditional storage systems. The basic server information and device parameters recorded by the root and leaf nodes can directly provide structured data support for resource status analysis, eliminating the need for additional time to sort out the hardware-data relationship. The hierarchical mapping relationship also makes subsequent partitioning and sub-region identification more targeted, avoiding inefficient waste caused by indiscriminate data processing. Partitioning units are divided based on data distribution density and business relevance, and sub-regions are divided using cluster analysis, enabling data to be clustered according to characteristics, avoiding resource scheduling chaos caused by mixed storage of data with different densities and different businesses. Data is mapped to sub-regions based on physical location and access characteristics, so that data within the same sub-region has similar storage location attributes and access patterns. Subsequent resource adjustments can be made precisely based on the characteristics of the sub-regions. For example, high I / O performance resources can be centrally configured for high-frequency access sub-regions to improve the targeting of resource allocation.

[0037] In a preferred embodiment of the present invention, step 1.3, based on the distribution density and business relevance of multimodal data in the multidimensional data space, divides the tree-like abstract data structure into regions, forming multiple partition units with physical server nodes as basic units, including: Step 1.3.1: Based on the hierarchical relationship between physical server nodes and storage devices in the tree-like abstract data structure, define the radial dimension for measuring the distribution density of data on physical storage devices and the axial dimension for distinguishing different rack server node data types, and construct a cylindrical coordinate mapping model. Specifically, this includes: based on the hierarchical relationship of physical server nodes as root nodes and storage devices as leaf nodes in the tree-like abstract data structure, determining the data flow path and ownership logic in the hardware architecture.

[0038] The radial dimension is defined to measure the distribution density of data on physical storage devices. This dimension is centered on a single physical server node and extends to each storage device directly associated with it. The density of data distribution is quantified by the number of data distributed on different storage devices and their storage proportion.

[0039] An axial dimension is defined to distinguish the data types of different rack server nodes. This dimension is based on the rack and is classified and identified according to the business type of the data processed by the physical server nodes within the rack (such as database business data, office application data, and AI training data), so as to achieve a clear division of the data types of different rack server nodes.

[0040] By combining radial and axial dimensions, a cylindrical coordinate mapping model is constructed. The axis of the cylinder corresponds to the axial dimension, which is used to represent the data type classification of the rack and server nodes. The radius of the cylinder corresponds to the radial dimension, which is used to represent the data distribution density on the storage device. This model transforms the data distribution characteristics in the multidimensional data space into intuitive geometric coordinate relationships.

[0041] Step 1.3.2: Based on the cylindrical coordinate mapping model, map each data point in the multidimensional data space to the corresponding cylindrical coordinates according to its physical server node and storage device partition. Then, perform initial division of the data according to the axial dimension to form several candidate partitions based on physical server nodes. Specifically, this includes: based on the constructed cylindrical coordinate mapping model, sorting out the physical server node and storage device partition information corresponding to each data point in the multidimensional data space.

[0042] Each data point is mapped to a corresponding position in cylindrical coordinates according to the identifier of its physical server node and the specific partition location of the storage device associated with that physical server node. The coordinates of the data point in the axial dimension are determined by the data type of the rack and server node to which it belongs, and the coordinates in the radial dimension are determined by the distribution density of the storage device where the data is located.

[0043] Based on the classification logic of the axial dimension, all data points after mapping are initially divided, and data points belonging to the same physical server node are grouped together to form several candidate partitions based on a single physical server node. Each candidate partition contains all data points on the storage device associated with that physical server node, ensuring that the initial partition is consistent with the hardware boundary of the physical server node.

[0044] Step 1.3.3: For each candidate partition, calculate the distribution density of its internal data points at the physical storage device level to obtain the adjusted partitions. Specifically, this includes: for each candidate partition, extracting the storage device information corresponding to all data points in the partition, including the storage device number where the data points are located, the physical partition, and the storage capacity ratio.

[0045] The distribution of data points across each associated storage device within each candidate partition is counted. The average distribution of data points on a single storage device is calculated. Simultaneously, the total storage capacity and used capacity of the storage devices are combined to convert the data distribution density value at the storage device level. A higher density value indicates that the data distribution on that storage device is more dense.

[0046] Based on the calculated distribution density value, the candidate partitions are adjusted. If the distribution density of some storage devices in a candidate partition far exceeds the preset threshold, it indicates that the data in that area is over-concentrated. The data points on the over-concentrated storage devices need to be split into the areas corresponding to storage devices with lower distribution density under the same physical server node to form the adjusted partitions. If the distribution density is within a reasonable range, the original range of the candidate partitions remains unchanged.

[0047] Step 1.3.4: For each adjusted partition, merge and optimize it based on the business relevance between its data attributes and the topological proximity of the physical server nodes to form multiple partition units with physical server nodes as the basic unit. Specifically, this includes: extracting the attribute information of the data within each adjusted partition, including the business task identifier to which the data belongs, the data access mode and the data processing requirements, and analyzing the business relevance between the data attributes of different partitions. For example, data belonging to the same database business has high business relevance, while data belonging to office applications and AI training have low business relevance.

[0048] At the same time, the topology location information of the physical server nodes corresponding to each partition after adjustment is obtained, and the physical proximity of the server nodes in the rack is determined. The higher the proximity, the closer the hardware connection distance between the server nodes and the lower the data interaction latency.

[0049] Based on business relevance and topological proximity, the merging optimization is performed. For multiple adjusted partitions with high business relevance and corresponding physical server node topological proximity, if the overall data distribution density after merging is still within a reasonable range and will not exceed the resource carrying capacity of a single physical server node, they are merged into one partition unit. For adjusted partitions with low business relevance or poor topological proximity, they remain independent partitions. Through the above merging optimization, multiple partition units with physical server nodes as the basic unit are finally formed, ensuring that the data business attributes within the partition unit are unified and the hardware topology is centralized.

[0050] In this embodiment of the invention, based on the hierarchical relationship between physical server nodes and storage devices in a tree structure, a model is constructed by defining radial (data distribution density) and axial (data type) dimensions. This transforms the abstract data distribution of the multidimensional data space into an intuitive geometric coordinate relationship, avoiding the problem of data characteristics being disconnected from physical storage in traditional partitioning methods, and ensuring a deep match between data partitioning and hardware architecture.

[0051] Based on the cylindrical coordinate mapping model, data points are mapped to physical server nodes and storage devices, and candidate partitions are initially defined using the axial dimension. Each candidate partition directly corresponds to a single physical server node, adhering to the principle of using physical server nodes as the basic unit to avoid data mixing across server nodes and reduce the complexity of subsequent adjustments. The data distribution density at the physical storage device level is calculated for each candidate partition, and the partition range is adjusted based on the density value. This effectively separates partitions with excessively concentrated data and supplements sparse data partitions, preventing some storage devices from being overloaded or idle due to uneven data distribution, thus improving the balance of data distribution within each partition. The adjusted partitions are then merged based on data business relevance and server node topological proximity, ensuring that data within the same partition unit has unified business attributes and that server nodes are physically close. On one hand, consistent business relevance reduces interference from cross-business data scheduling and improves the efficiency of business-scenario-based resource allocation; on the other hand, topological proximity shortens data interaction distances, reduces cross-node data transmission latency, and further improves the overall performance and resource utilization efficiency of the storage system.

[0052] In a preferred embodiment of the present invention, step 1.4, performing cluster analysis on the data distribution within each partition unit, dividing it into sub-regions, and mapping each data instance to its respective sub-region based on its physical location and access characteristics, includes: Step 1.4.1: For each partition unit with physical server nodes as the basic unit, extract the spatial coordinates, physical location attributes, and access feature attributes of all data points within it to form the dataset to be clustered. Specifically, this includes: for each partition unit with physical server nodes as the basic unit, extracting three types of core features of the data points from the multidimensional data space and tree-like abstract data structure: Spatial coordinates: Extract the radial coordinates (reflecting data distribution density) and axial coordinates (reflecting data type) of the data points in the cylindrical coordinate mapping model. The coordinate values ​​must be consistent with the coordinate range of the partition unit in step 1.3. Physical location attributes: Include the physical partition identifier of the storage device where the data point is located, the location of the storage server rack slot, and the connection interface number between the storage device and the server, ensuring that they correspond to the physical dimensions of the multi-dimensional data space constructed in step 1.1. Access characteristic attributes: Extracted from historical data, covering the access frequency of the data point (number of accesses in the past hour), access time distribution (e.g., morning peak 8-10 am, evening peak 18-20 pm), read / write operation ratio (number of read operations / number of write operations), and average access response time. The data must be tested for outliers by IQR to remove invalid records.

[0053] The extracted three types of features are organized according to the structure of unique identifier of data point, spatial coordinates, physical location attribute and access feature attribute to form feature record of each data point. Among them, the unique identifier of data point adopts partition unit coding and data block ID rules (such as PU-R03-S08-001), spatial coordinates are retained to three decimal places, physical location attribute is coded according to rack, server, device and partition level (such as R03-S08-SSD01-P003), and numerical data in access feature attribute are retained to two decimal places. After all feature records are summarized, they are stored as a dataset to be clustered.

[0054] Step 1.4.2: Based on the physical location distribution characteristics of the data points in the dataset to be clustered, obtain its initial convex hull; traverse each edge of the initial convex hull and calculate the concavity depth of the data points in its neighborhood; according to a preset concavity depth threshold, identify concave points that meet the conditions and add them to the boundary point set until no new concave points that meet the conditions are generated, so as to obtain a concave polygon constructed from the boundary point set. Specifically, this includes: generating the initial convex hull using the Graham scan algorithm based on the physical location attributes of the data points in the dataset to be clustered. First, the extreme points of the physical location attributes of the data points are selected, including the data points with the front / backmost rack slot positions and the smallest / largest storage device partition numbers, as the candidate set of convex hull vertices; then, the candidate points are sorted according to the polar angle of the physical location coordinates, and adjacent points are connected in sequence to form a closed polygon, ensuring that the polygon can completely cover the physical distribution range of all data points in the partition unit, thus obtaining the initial convex hull. Traverse each edge of the initial convex hull, using the two endpoints of each edge as a reference to determine the neighborhood range of the edge (the neighborhood width is set to 1 / 5 of the edge length, dynamically adjusted according to the physical partition size of the storage device); calculate the vertical distance from all data points in the neighborhood to the edge, this distance is the concavity depth, the larger the distance, the more significantly the data point deviates from the convex hull contour; preset the concavity depth threshold, the threshold is set to 1.2 times the average distance between the physical positions of all data points in the partition unit (e.g., if the average distance is 0.5 units, the threshold is set to 0.6 units); identify data points with a concavity depth greater than the threshold as concave points, add them to the boundary point set, and update the convex hull contour so that the new contour includes the concave point; repeat the above traversal, calculation and identification process until no new concave point meets the condition after traversing all convex hull edges, at this time the points in the boundary point set are connected in order of physical position to form a concave polygon that highly matches the actual physical distribution of the data.

[0055] Step 1.4.3: Using the concave polygon as the constraint boundary, perform cluster analysis on the dataset to be clustered, grouping data points located within the same concave polygon, physically adjacent, and with similar access characteristics into the same sub-region. Specifically, this includes: using the concave polygon obtained in Step 1.4.2 as the spatial constraint boundary to ensure that the clustering result does not cross the physical storage area corresponding to the concave polygon (e.g., if a concave polygon corresponds to physical partitions 1-3 of the storage device, then the clustering is only performed within that partition), avoiding data mixing between different physical partitions; First stage: clustering by physical proximity, calculating the physical distance between each data point in the dataset to be clustered and the vertex of the concave polygon, grouping data points whose distance is less than 0.8 times the average physical distance between partition units into the same preliminary cluster group, ensuring that the physical distribution of data points within the group is concentrated.

[0056] The second stage involves optimization based on access feature similarity. The similarity of access features among data points within the initial cluster group is calculated (e.g., a difference in access frequency of less than or equal to 5 times / hour, a difference in read / write ratio of less than or equal to 0.2, and a difference in response time of less than or equal to 10 milliseconds are considered similar). Data points with dissimilar access features are separated from the group and reassigned to cluster groups with more matching access features. This ultimately forms multiple sub-regions, each satisfying three conditions: being located within the same concave polygon, being physically adjacent, and having similar access features. The number of sub-regions is dynamically adjusted based on the number of concave polygons and the data distribution density. For example, if data is dense within a single concave polygon, it is split into 2-3 sub-regions; if data is sparse, one sub-region is retained.

[0057] Step 1.4.4: For each sub-region obtained by clustering, calculate the physical location center and access feature center of its data points, and map each data instance in the partition unit to its corresponding sub-region based on its physical location similarity and access feature similarity. Specifically, this includes: collecting the physical location attributes of all data points in the sub-region, extracting key parameters such as rack slot location and storage device partition number, and summing the center value according to (each parameter value × corresponding weight) (rack slot location weight 0.6, partition number weight 0.4, because the slot location has a greater impact on the data access path), to obtain the physical location center of the sub-region; Access feature center: Collect access feature attributes of all data points in the sub-region, and calculate the center value using the weighted average method (access frequency weight 0.3, read / write ratio weight 0.2, response time weight 0.5, since response time directly reflects the urgency of resource demand), to obtain the access feature center of the sub-region.

[0058] Calculate the similarity between each data instance and each sub-region within the partition unit. The physical location similarity is calculated as 1 - (distance between the physical location center of the data instance and the sub-region ÷ maximum physical location distance of the partition unit). The closer the result is to 1, the better the location match. The access feature similarity is calculated as 1 - (sum of absolute values ​​of the differences between the access feature centers of the data instance and the sub-region ÷ maximum possible value of the feature difference). The closer the result is to 1, the better the feature match.

[0059] The overall similarity is calculated as physical location similarity × 0.4 + access feature similarity × 0.6, and the data instance is mapped to the sub-region with the highest overall similarity. If there are cases with equal overall similarity, the access feature similarity is given priority (because access features are directly related to resource requirements). After the mapping is completed, the sub-region identifier of each data instance is updated through the metadata management module to ensure that the sub-region data can be accurately located when extracting features in the subsequent step 1.5, so as to provide an accurate data foundation for the calculation of resource weight coefficients.

[0060] In this embodiment of the invention, a dataset is constructed by extracting the spatial coordinates, physical location attributes, and access feature attributes of data points within a partition unit. The data undergoes outlier detection processing to ensure data integrity and accuracy. The spatial coordinates are associated with a pre-constructed cylindrical coordinate model, the physical location attributes correspond to the hardware topology, and the access feature attributes reflect actual usage requirements. These three types of features comprehensively cover the core attributes of the data, avoiding clustering bias due to missing features. An initial convex hull is generated based on the physical location distribution of the data points. Concave polygons are constructed by combining concave depth calculation and concave point recognition. Compared to traditional convex hulls that only wrap the outer contour of the data, concave polygons better match the actual physical distribution of the data, accurately reflecting the dense and sparse areas of data within the physical partition of the storage device. This boundary division, highly matched to the hardware physical architecture, prevents subsequent clustering from crossing physical partitions.

[0061] In this embodiment, concave polygons are used as constraint boundaries, and clustering is performed by combining physical proximity and access feature similarity to ensure that data within the same sub-region is both physically concentrated and has similar access patterns. For example, data that are physically located in the same storage device partition and are frequently accessed are grouped into the same sub-region to avoid mixing data with different access needs. Subsequent resource scheduling can be precisely configured according to the characteristics of the sub-region (such as prioritizing high I / O performance resources for high-frequency access sub-regions), improving the targeting of resource allocation and solving the scheduling inefficiency problem caused by traditional clustering that emphasizes location over needs. By calculating the physical location center and access feature center of the sub-region and combining comprehensive similarity, data instances are mapped to the corresponding sub-regions to ensure that each data instance is accurately assigned. This mapping method considers both the physical storage location of the data and the actual access needs. When extracting sub-region features in the subsequent step 1.5, the data can be accurately located, and the calculated resource weight coefficients are more in line with the actual needs. This allows the resource adjustment parameters to effectively guide scheduling decisions, reduce resource waste caused by data mismatch, and improve the overall scheduling efficiency and service quality of the storage system.

[0062] In a preferred embodiment of the present invention, step 2, which involves concatenating the feature vectors of different modalities to obtain a multimodal feature vector, includes: Textual data is extracted from the business logs, user task submission interface, and data metadata annotation module of the distributed storage system. Specifically, this includes business task descriptions (such as AI large model training dataset storage requests and OLTP database real-time read / write task instructions), data usage annotations (such as hot data high-frequency access identifiers and cold data archive storage identifiers), and user requirements for storage services (such as low-latency storage requirements and high-reliability storage requirements).

[0063] First, preprocess the text data by converting all text to lowercase and removing meaningless special characters (such as commas, semicolons, and asterisks) and stop words (such as "of", "used for", and "to carry out"). Then, according to the input requirements of the BERT model, truncate or pad the text to 512 characters (the standard input length of the BERT basic model) and add the [CLS] identifier at the beginning of the text and the [SEP] identifier at the end of the text.

[0064] The preprocessed text is input into a pre-trained BERT base model (such as the bert-base-uncased version). The model performs semantic analysis on the text through a 12-layer Transformer encoder. The bottom encoder captures surface features such as part-of-speech and position of words, while the upper encoder mines implicit business relationships in the text (such as the implicit connection between AI training and high I / O requirements). The vector corresponding to the [CLS] identifier in the model's output layer is extracted. This vector is the text feature vector, with a fixed dimension of 768 (the default output dimension of the BERT base model). Each value represents the feature weight of the text in a certain semantic dimension; the larger the value, the stronger the association between the text and that semantic dimension. The 768-dimensional text feature vector is standardized to adjust all values ​​to the range of 0-1, avoiding excessively high weights for any feature after subsequent concatenation due to differences in numerical scale, and ensuring a balanced influence of text features on other modal features.

[0065] A solution for acquiring real-time hardware operation data involves collecting continuous time-series data from monitoring modules of compute nodes, storage nodes, and network devices (such as Prometheus and NodeExporter). Compute node data: CPU utilization, GPU utilization, collected every 1 second; Storage node data: SSD and HDD IOPS, read / write throughput, average response time, collected every 1 second; Network device data: Bandwidth usage, data transmission latency, packet loss rate, collected every 1 second; Time series windows are divided according to business scenario requirements: For scenarios sensitive to real-time performance, such as AI training and OLTP, a 1-minute time series window (containing 60 data points) is used; For scenarios such as office applications and cold data storage, a 5-minute time series window (containing 300 data points) is used. The length of the time series window can be dynamically adjusted according to business needs.

[0066] The LSTM time series model is used to process the time series data. The continuous data within each time window is arranged into a one-dimensional data sequence in chronological order. For example, the CPU utilization data sequence for a 1-minute window contains 60 consecutive utilization values. The data sequence is input into an LSTM model. The model analyzes the data trend through memory units and gating mechanisms (input gate, forget gate, output gate). The forget gate filters out meaningless normal fluctuation data, the input gate retains key change data (such as a sudden spike in GPU utilization), and the output gate outputs the trend characteristics of the time series data. The model outputs a time series feature vector with the same number of data points as the time series window. For example, a 1-minute window corresponds to a 60-dimensional vector, and a 5-minute window corresponds to a 300-dimensional vector. Each value represents the trend characteristic value of the time series data within the corresponding time slice (e.g., a high value indicates that the I / O performance is at its peak within that time slice).

[0067] Using the same standardization method as text feature vectors, the values ​​of time-series feature vectors are mapped to the 0-1 range to ensure numerical scale consistency with text feature vectors. From the constructed multidimensional data space and tree-like abstract data structure, the hardware physical attributes corresponding to the data are extracted, specifically including: Storage server related information: rack number, server slot position in the rack, number of server CPU cores, total server memory capacity; Storage device related information: device type (SSD set to 1, HDD set to 0), number of physical partitions, capacity of each partition, device connection interface type (PCIe set to 1, SATA set to 0); Operating status related information: current device temperature, remaining storage space, percentage of used storage space.

[0068] Based on the categories and number of hardware physical attributes, the device physical feature vector is determined to have 32 dimensions, with each attribute category corresponding to 1 dimension: Non-numerical attributes (such as rack number and interface type) are numerically encoded, for example, rack number R01 is set to 1, R02 is set to 2, PCIe interface is set to 1, and SATA interface is set to 0; numerical attributes (such as partition capacity and remaining storage space) are normalized, for example, when the maximum partition capacity is 200GB, 50GB corresponds to 0.25 and 100GB corresponds to 0.5; all the encoded physical attribute values ​​are filled into the corresponding dimensions of a 32-dimensional vector in the order of rack, server, storage device, and operating status to form a device physical feature vector. The values ​​of this vector are already in the range of 0-1 and do not require additional standardization.

[0069] Check if the dimensions of the three types of modal feature vectors meet the preset standards: text feature vectors must be 768-dimensional, temporal feature vectors must be the dimensions corresponding to the preset window (60-dimensional or 300-dimensional), and device physical feature vectors must be 32-dimensional. If the temporal feature vector has a dimensional deviation due to window adjustment (e.g., actually 45-dimensional), pad it to the target dimension (e.g., pad to 60-dimensional) using zero-padding to ensure that each vector dimension is compliant. Check if all values ​​of the three types of vectors are within the range of 0-1. If there are outliers outside the range (e.g., 1.2 due to BERT model output deviation, negative values ​​output by LSTM model), truncate the values ​​to correct them to the range of 0-1 (negative values ​​are set to 0, and values ​​greater than 1 are set to 1) to avoid outliers affecting the validity of the concatenated vectors.

[0070] The concatenation order is determined as device physical feature vector, temporal feature vector, and text feature vector, specifically: First, a 32-dimensional physical feature vector of the device is concatenated, as it is directly related to the hardware topology and provides a physical spatial basis for subsequent scheduling decisions. Then, a time-series feature vector (60-dimensional or 300-dimensional) is concatenated. The time-series features reflect the real-time operating status of the resources, and the adjacent physical features can reflect the relationship between physical location and real-time status. Finally, a 768-dimensional text feature vector is concatenated. The text features provide business semantic information, which is used for subsequent user satisfaction prediction and business priority classification.

[0071] The matrix concatenation algorithm is used to concatenate the three types of vectors into a single array in sequence: Example 1 (1-minute time series window): 32-dimensional device physical feature vector + 60-dimensional time series feature vector + 768-dimensional text feature vector, forming an 860-dimensional multimodal feature vector; Example 2 (5-minute time series window): 32-dimensional device physical feature vector + 300-dimensional time series feature vector + 768-dimensional text feature vector, forming an 1100-dimensional multimodal feature vector.

[0072] The concatenated multimodal feature vectors are associated with unique identifiers for data points (such as partition unit codes-data block IDs) and stored in the feature vector database in key-value pair format. Key: Uses a unique identifier for the data point and a splicing timestamp rule (e.g., PU-R03-S08-001-202409091000), with the timestamp accurate to the minute, to distinguish vectors of the same data point in different time windows; Value: Stores a floating-point array of multimodal feature vectors, along with a splicing log (including the original dimension, standardization method, and splicing order of each modality vector).

[0073] Following the input format of the Random Forest algorithm, the multimodal feature vectors are converted into a two-dimensional feature matrix, with each row corresponding to a data point vector and each column corresponding to a feature dimension, for the algorithm to predict user satisfaction. According to the requirements of reinforcement learning algorithms (DQN, PPO), the device physical features and temporal features in the vectors are extracted as state space inputs, and the text features are used as business weight inputs to meet the model calculation requirements. If a certain modality feature vector needs to be regenerated due to data updates (such as time window scrolling or text data addition), the feature splicing module automatically triggers re-sponsoring and updates the stored multimodal feature vectors.

[0074] In a preferred embodiment of the present invention, step 3, based on multimodal feature vectors and resource adjustment parameters, predicts user satisfaction; and based on user satisfaction, establishes a resource demand priority mapping table for different business scenarios, including: Step 3.1 involves weighted feature fusion of the multimodal feature vectors and resource adjustment parameters to obtain a fused feature vector. Specifically, this includes establishing a one-to-one correspondence between the multimodal feature vectors and resource adjustment parameters, using the unique identifier of each data point as an index. The multimodal feature vectors are derived from Step 2 and include device physical feature vectors (32-dimensional), temporal feature vectors (dynamic dimension), and text feature vectors (768-dimensional). The resource adjustment parameters are derived from Step 1.5, i.e., the resource weight coefficients of the sub-regions. This ensures that each fusion object is associated with data from the same sub-region, avoiding cross-sub-region data mixing that could affect the fusion accuracy.

[0075] Weighting coefficients are assigned based on differences in business scenarios, and the sum of the weighting coefficients for all scenarios is 1. In the AI ​​large-scale model training scenario, the multimodal feature vector weight is 0.6, and the resource adjustment parameter weight is 0.4. This scenario relies on multimodal data to accurately identify high computing power and high I / O requirements, while the resource weight coefficient reflects the resource scarcity of sub-regions.

[0076] In an OLTP database scenario, the weight of the multimodal feature vector is 0.5, and the weight of the resource adjustment parameter is also 0.5. In this scenario, business semantics, temporal features, and resource usage have an equally important impact on satisfaction.

[0077] In an office application scenario, the multimodal feature vector weight is 0.4, and the resource adjustment parameter weight is 0.6. The business requirements in this scenario are relatively simple, and resource usage has a more significant impact on user satisfaction.

[0078] In cold data storage scenarios, the multimodal feature vector weight is 0.3, and the resource adjustment parameter weight is 0.7. In this scenario, data access frequency is low, and storage capacity requirements are the core factor affecting satisfaction; therefore, the resource weight coefficient has a higher proportion.

[0079] The fusion is achieved using element-level operations. First, the resource adjustment parameters (single values) are expanded into vectors of the same dimension as the multimodal feature vector. Then, each value of the multimodal feature vector is multiplied by its corresponding weighting coefficient. The expanded resource adjustment parameter vector is also multiplied by its corresponding weighting coefficient. Finally, the two sets of results are added element-wise to obtain a fused feature vector with the same dimension as the multimodal feature vector. For example, if the multimodal feature vector is 860-dimensional (32 + 60 + 768), the expanded resource adjustment parameter vector is also 860-dimensional. The two sets of weighted results are added together to generate an 860-dimensional fused feature vector.

[0080] Step 3.2: Obtain historical resource usage data and real-time monitoring data from compute nodes, storage nodes, and network devices in the distributed storage system, and perform standardization processing to extract time-series and statistical features related to physical resource distribution. Specifically, this includes collecting two types of data using Prometheus and NodeExporter: Historical resource usage data is extracted from the historical database of the storage system for the past 30 days. This includes the daily average and peak values ​​of CPU / GPU utilization of compute nodes, daily statistics of I / O performance of SSD / HDD of storage nodes (average IOPS, maximum throughput, average response time), and daily peak bandwidth usage and average latency of network devices. The collection granularity is 1 hour / data point to ensure coverage of long-term resource usage patterns.

[0081] Real-time monitoring data is extracted from the monitoring modules of each node, with a collection frequency consistent with step 1 (1 second / time for AI scenarios and 5 seconds / time for office scenarios). This includes real-time CPU / GPU load of computing nodes, real-time I / O response time of storage nodes, and real-time bandwidth usage and packet loss rate of network devices, capturing short-term dynamic changes in resources.

[0082] The IQR method is used to detect outliers in two types of data, such as CPU utilization greater than 100%, SSD response time exceeding the normal range by 3 times, and sudden increases in network latency. Outliers are replaced with the median of the time period in which the data occurred to avoid interference from outliers in feature extraction. The Z-score method is used to standardize the processed historical and real-time data, mapping the values ​​to the range of -1 to 1. The historical data is calculated based on the 30-day mean and standard deviation, while the real-time data is calculated based on the mean and standard deviation of the past hour, ensuring a consistent scale for resource data of different time spans and types, and meeting the input requirements of subsequent models.

[0083] Two types of features are extracted from the standardized data: The time-series features extract trend characteristics from historical resource data, such as the weekly fluctuation cycle of CPU utilization and the monthly slope of SSDI / O performance; and extract short-term fluctuation characteristics from real-time data, such as the change in GPU utilization in the past 10 minutes and the fluctuation in network bandwidth in the past 5 minutes. The time-series features are fixed at 40 dimensions, with 30 dimensions from historical data and 10 dimensions from real-time data. The statistical features include calculating the mean, variance, maximum, minimum, median, quartile Q1, quartile Q3, and range of historical data, totaling eight statistical measures. The 5-minute mean, 10-minute peak, and 1-hour mean of real-time data are also calculated and added to the statistical features. The final statistical feature dimension is fixed at 11 dimensions, comprehensively reflecting the global statistical attributes of the resource data.

[0084] Step 3.3 involves concatenating the fused feature vector with the temporal and statistical features to form the input feature set. Specifically, this includes checking the dimensions of the fused feature vector, temporal features, and statistical features. The dimensions of the fused feature vector vary with the business scenario's temporal window (e.g., 860 dimensions for a 1-minute window and 1100 dimensions for a 5-minute window). The temporal features are fixed at 40 dimensions, and the statistical features are fixed at 11 dimensions. There is no need to adjust the dimensions of each feature; it is only necessary to ensure that the values ​​of all features are within the standardized range (-1 to 1 or 0 to 1). If there are values ​​that exceed the range, they are corrected to the corresponding range through truncation.

[0085] First, feature vectors are concatenated and fused, which contain core features of data association and form the basis for predicting user satisfaction. Then, 40-dimensional time-series features are concatenated to supplement the time trend information of resource data and improve the adaptability of the prediction model to dynamic changes in resources. Finally, 11-dimensional statistical features are concatenated to provide global statistical attributes of resource data and assist the model in judging the overall supply and demand of resources. The three types of features are combined in sequence through feature concatenation to form a single input feature record. All input feature records are summarized to form an input feature set. Each record is accompanied by a business scenario identifier associated with the data (such as AI large model training scenario identifier S-AI, database OLTP scenario identifier S-DB-OLTP).

[0086] Step 3.4: Based on the input feature set, construct a prediction model; according to the prediction model, obtain the predicted value of user satisfaction, specifically including: the number of decision trees is set to 100, which is determined based on previous experimental verification: when the number of decision trees is less than 80, the model is prone to underfitting (the accuracy of the test set is less than 85%); when there are more than 120 trees, the calculation time increases by 30% but the accuracy is improved by less than 2%; 100 trees can achieve the optimal balance between accuracy and efficiency.

[0087] The maximum depth of each decision tree is adjusted according to the dimension of the input feature set, following the adaptation rule that dimension × 0.055 ≈ maximum depth. For example, 911-dimensional features correspond to 50 (911 × 0.055 ≈ 50.1), and 860-dimensional features correspond to 47 (860 × 0.055 ≈ 47.3). At the same time, the minimum number of sample splits is set to 5 (i.e., no splitting occurs when the number of node samples is less than 5), and the minimum number of leaf nodes is set to 2 (to avoid overfitting caused by a single sample), further controlling the complexity of the decision tree. The node splitting criterion adopts the Gini coefficient because the Gini coefficient is more efficient than information gain, and in continuous value regression scenarios such as satisfaction prediction, it can more quickly divide the sample groups with similar satisfaction levels, adapting to the real-time prediction requirements of distributed storage systems.

[0088] To avoid overfitting, a random forest feature selection mechanism is enabled. When constructing each decision tree, only 70% of the input feature set is randomly selected for splitting (e.g., approximately 638 dimensions are randomly selected from 911-dimensional features), reducing the excessive influence of single-class features on the model. The out-of-bag (OOB) validation ratio is set to 30%, using out-of-bag samples that did not participate in decision tree training to evaluate model performance in real time without the need for additional validation sets, thus improving training efficiency. At the same time, a class weight balancing mechanism is enabled. Given that 60% of historical satisfaction ratings are 3-4 points, while only 20% are 1-2 points and 5 points, a 1.5x weight is assigned to minority class samples (1-2 points and 5 points) to ensure balanced prediction accuracy for each satisfaction level.

[0089] Stratified sampling is used to extract data from the input feature set to ensure that the satisfaction level distribution of the training and test sets is consistent with the original input feature set. For example, if 20% of the samples in the original data are 1-2 points, 60% are 3-4 points, and 20% are 5 points, then the training set (70% of the data) and the test set (30% of the data) should also maintain this proportion to avoid missing samples of a certain satisfaction level due to random sampling, which would affect the model's generalization ability.

[0090] After sampling, historical user satisfaction rating labels are matched for each input feature record: valid feedback from the past 3 months is extracted from the system's user feedback records and associated with business scenarios and resource usage periods. For example, in the AI ​​large model training scenario, user feedback on using resources in a certain sub-region from 9-10 am on August 10, 2024, is matched with the input feature labels of the data in that sub-region during that period. The rating is divided into 1-5 points, where 5 points means fully meeting the requirements, 4 points means basically meeting the requirements, 3 points means average, 2 points means somewhat dissatisfied, and 1 point means completely dissatisfied, ensuring that the labels are completely aligned with the time, scenario, and resource association dimensions of the input features.

[0091] The matched training data is cleaned by removing input feature records with missing labels (less than 1%) and records with null values ​​in the input features (null values ​​that can be repaired are filled in by interpolation of the preceding and following data; those that cannot be repaired are directly removed), ultimately ensuring that the integrity of the training set data reaches over 99%. The cleaned training and test sets are then converted to a model-adapted format. The input feature part is converted into a floating-point two-dimensional array (each row corresponds to one record, and each column corresponds to one feature dimension), and the label part is converted into a one-dimensional floating-point array. At the same time, the training data is divided into batches (128 records per batch) to meet the needs of batch training of the model and improve training speed.

[0092] The first stage is initial training, where the first batch of training data (20% of the total training set) is input into the model, and initial training is carried out with a learning rate of 0.1 and 50 iterations. At this time, the model only learns the correlation between basic features and satisfaction (such as high I / O performance features corresponding to high satisfaction), and the mean squared error is controlled within 0.3.

[0093] The second stage is refined training. The remaining 80% of the training data is input, the learning rate is adjusted to 0.05, and the number of iterations is 100. After each iteration, the mean squared error of the training set and the out-of-bag error are calculated. If the mean squared error decreases by less than 0.001 after five consecutive iterations, the model is considered converged. If it has not converged and the out-of-bag error increases, it indicates overfitting, and the current iteration is immediately stopped, backtracking to the model parameters of the iteration with the lowest error. Throughout the training process, the splitting node features and Gini coefficient changes of each decision tree are recorded in real time, generating a training log. This facilitates tracing key influencing features during subsequent model optimization (e.g., the I / O delay feature in time-series features contributes the most to satisfaction prediction).

[0094] Input the test set into the trained model, calculate the mean squared error and accuracy between the predicted values ​​and the true labels. The mean squared error should be less than 0.1 and the accuracy should be no less than 90% (accuracy is calculated as the number of samples where the difference between the predicted value and the true label is less than or equal to 0.5 points / the total number of samples in the test set). Split the test set according to business scenarios and verify the model accuracy in AI large model training, database OLTP, office applications, and cold data storage scenarios respectively, ensuring that the accuracy in each scenario is no less than 88% to avoid the model being biased towards one scenario, resulting in insufficient prediction accuracy in other scenarios. If the verification does not meet the standards, optimize according to the following priority: First, increase the number of decision trees (20 trees each time, up to a maximum of 150 trees); if the accuracy still does not improve, adjust the maximum depth of the decision trees (fine-tuning in increments of ±5); finally, optimize the feature random selection ratio (adjusting between 60% and 80%). Retrain and verify after each optimization until all verification indicators meet the standards.

[0095] New data (i.e., real-time generated resource feature data) outside the test set is filtered from the input feature set. This data is then associated with individual records and scene identifiers and input into the model. The model performs parallel computation using 100 decision trees: each decision tree traverses the split nodes based on the input features and outputs the corresponding satisfaction prediction score (1-5 points); after all decision trees have been computed, the arithmetic mean of the 100 prediction scores is taken, and one decimal place is retained as the final prediction value (e.g., 4.2 points, 2.8 points); during the prediction process, the distribution of prediction scores for each decision tree is recorded. If the standard deviation of the 100 prediction scores for a certain data point is greater than 0.8 (indicating a large difference in predictions between decision trees), it is marked as data to be reviewed and will be subject to further verification; the predicted value of a certain data point is calculated and averaged with the predicted values ​​of other data in the same business scenario within the past 10 minutes. If the deviation exceeds 1 point (e.g., the average predicted value for the AI ​​scenario in the past 10 minutes is 4.1 points, and the predicted value for this data point is 2.9 points), an anomaly investigation is triggered.

[0096] Verify that the feature concatenation order is correct, that the standardized values ​​are within the range of -1 to 1 or 0 to 1, and that the physical resource features match the actual hardware configuration (e.g., whether the GPU memory is correctly labeled). Then check the model input format to ensure there are no missing dimensions or incorrect data types. After identifying and correcting anomalies, re-enter the model for prediction. If the deviation still exceeds 0.5 points, trigger temporary model optimization (e.g., increase the weight of the sample in this scene) to ensure the consistency and accuracy of the final prediction results.

[0097] Step 3.5: Based on the predicted user satisfaction value, and combined with the preset business scenario classification tags and physical resource constraints, establish a resource demand priority mapping table for different business scenarios. The mapping table includes at least a scenario identifier, satisfaction level, corresponding resource priority weight, and required physical resource type. Specifically, it includes four preset tags: AI large model training scenario tag S-AI, database OLTP scenario tag S-DB-OLTP, office application scenario tag S-OFF, and cold data storage scenario tag S-COLD. Each tag corresponds to a unique code to facilitate scenario identification. Physical resource constraints: AI large model training scenarios require GPU computing nodes, NVMe SSDs, and 10 Gigabit network devices; database OLTP scenarios require high IOPS SSDs, multi-core CPU nodes, and low packet loss rate network devices; office application scenarios require ordinary CPU nodes, SATA SSDs or HDDs, and gigabit network devices; cold data storage scenarios require low-cost HDDs and ordinary CPU nodes, without high bandwidth resources.

[0098] User satisfaction predictions (1.0-5.0 points) are divided into 5 levels, each corresponding to a fixed resource priority weight. Higher weight values ​​indicate more urgent resource needs: Extremely high satisfaction (5.0-4.5 points, resource priority weight 0.2): Current resource configuration fully meets needs; maintain the existing configuration without prioritizing additional resources. High satisfaction (4.4-3.5 points, resource priority weight 0.4): Resource configuration basically meets needs; supplement only when system resources are sufficient; priority lower than medium-high and above levels. Moderate satisfaction (3.4-2.5 points, resource priority weight 0.6): Slight resource shortages (e.g., occasional latency); medium-high priority required, prioritizing supplementation of core resources (e.g., storage I / O resources). Low satisfaction (2.4-1.5 points, resource priority weight 0.8): Resource shortages (e.g., frequent lag, I / O bottlenecks); high priority required, prioritizing allocation of sufficient resources to improve business experience. Extremely low satisfaction (1.4-1.0 points, resource priority weight 1.0). If resources are severely insufficient (e.g., there is a risk of business interruption), they should be given the highest priority, and idle system resources should be immediately allocated or low-priority business resources should be adjusted to supplement them.

[0099] The mapping table structure is built using fields for scene identifier, satisfaction level, resource priority weight, and physical resource type. The physical resource type needs to be refined based on scene constraints and satisfaction level. For example, in the S-AI scene, extremely low satisfaction (1.4-1.0 points) corresponds to physical resource types such as GPU compute nodes (≥24GB VRAM), NVMe SSDs (≥3500MB / s throughput), and 25G network devices (≤0.5ms latency); high satisfaction (4.4-3.5 points) corresponds to physical resource types such as GPU compute nodes (≥16GB VRAM), NVMe SSDs (≥3500MB / s throughput), and 25G network devices (≤0.5ms latency). MeSSD (throughput greater than or equal to 2500MB / s) and 10 Gigabit network devices (latency less than or equal to 2ms); store the mapping table in a structured format (such as JSON) to the configuration database, establish a joint index of scenario identifier and satisfaction level, and ensure that the query response time is less than or equal to 100ms; adjust the physical resource type parameters every quarter according to the system hardware upgrade. When the deviation between the predicted satisfaction value and the actual feedback for a certain scenario exceeds 0.5 points for three consecutive times, recalibrate the weight correspondence, and verify the effectiveness through test scenarios after the update. It is only effective if the adjusted predicted satisfaction value increases by more than or equal to 0.3 points.

[0100] In this embodiment, multimodal feature vectors are weighted and fused with resource adjustment parameters. This retains the comprehensive description of business needs and hardware status by multimodal features (device physical, temporal, and textual) while incorporating the resource stress of sub-regions reflected by resource adjustment parameters, avoiding the limitations of a single feature dimension. Weighting coefficients are dynamically assigned to different business scenarios, enabling the fused feature vectors to accurately match the core needs of the scenario (e.g., AI scenarios emphasize multimodal features, while cold data scenarios emphasize resource parameters). Historical and real-time resource data are standardized, invalid data is removed through IQR outlier detection, and Z-score standardization unifies the data scale to avoid interference from outliers and scale differences in feature extraction. Simultaneously, temporal and statistical features are extracted to capture both long-term resource variation patterns (e.g., weekly fluctuations in CPU utilization) and global data attributes (e.g., mean and peak values), ensuring that features can comprehensively characterize the physical resource distribution status, providing high-quality input data for the prediction model, and reducing prediction bias caused by data quality issues.

[0101] A random forest algorithm is used to construct the prediction model. Fine-tuned parameter settings (such as 100 decision trees and dynamically adjusting tree depth) balance accuracy and efficiency. Stratified sampling and weight balancing ensure a reasonable distribution of training data. Phased training and multi-dimensional validation further optimize model performance, resulting in a small deviation between predicted values ​​and actual satisfaction levels (mean squared error less than 0.1% and accuracy greater than or equal to 90%). Real-time prediction and multi-level verification mechanisms ensure consistent and accurate prediction results. A mapping table is constructed by combining predicted satisfaction values, scene labels, and physical resource constraints to determine resource priorities and physical resource types corresponding to different scenes and satisfaction levels. This allows resource allocation to accurately match scene requirements (e.g., low satisfaction levels in AI scenarios require high-memory GPUs and high-bandwidth networks). The dynamic update mechanism of the mapping table allows for configuration adjustments based on hardware upgrades and business changes, ensuring that resource scheduling always aligns with actual needs, improving resource utilization and user satisfaction.

[0102] In a preferred embodiment of the present invention, step 4, constructing a multi-objective optimization model based on the resource demand priority mapping table, includes: Step 4.1: Parse the resource requirement priority mapping table to obtain the resource priority weights and required physical resource types under different business scenarios. Specifically, this includes: parsing the mapping table one by one according to the business scenario classification tags; for the four core scenarios of AI large model training scenario (S-AI), database OLTP scenario (S-DB-OLTP), office application scenario (S-OFF), and cold data storage scenario (S-COLD), extract two key pieces of information corresponding to different satisfaction levels under each scenario: resource priority weight (0.2-1.0), required physical resource type and specific parameters (such as GPU memory, SSD throughput, network latency).

[0103] The parsed information is organized according to the logical association of scene identifier, satisfaction level, resource priority weight, and physical resource type parameters. For example, in the AI ​​large model training scenario, extremely low satisfaction (1.4-1.0 points) corresponds to a resource priority weight of 1.0, and the physical resource type parameters are GPU computing node (greater than or equal to 24GB video memory), NVMeSSD (greater than or equal to 3500MB / s throughput), and 25G network device (less than or equal to 0.5ms latency); high satisfaction (4.4-3.5 points) corresponds to a resource priority weight of 0.4, and the physical resource type parameters are GPU computing node (greater than or equal to 16GB video memory), NVMeSSD (greater than or equal to 2500MB / s throughput), and 10 Gigabit network device (less than or equal to 2ms latency). The organized data is stored in a temporary parsing library.

[0104] Step 4.2: Construct a multi-objective decision-making hierarchical model with resource utilization, service quality, and energy consumption as optimization objectives, and calculate the initial weight vector for each optimization objective based on the resource priority weights and physical resource type constraints. Specifically, this includes: constructing the multi-objective decision-making hierarchical model. The target layer focuses on maximizing resource utilization, optimizing service quality, and minimizing energy consumption as core optimization objectives, covering resource scheduling in distributed storage systems. The criterion layer corresponds to the detailed indicators of each optimization objective. Resource utilization criteria include CPU utilization, memory utilization, storage device utilization, and network bandwidth utilization; service quality criteria include I / O response time, data access success rate, and service interruption frequency; and energy consumption criteria include computing node energy consumption, storage node energy consumption, and network device energy consumption. The solution layer outlines the resource scheduling direction corresponding to each criterion layer indicator. For example, resource utilization criteria correspond to load balancing scheduling, service quality criteria correspond to high-priority resource guarantees, and energy consumption criteria correspond to low-load node hibernation.

[0105] Combining the resource priority weights obtained from the analysis with the physical resource type constraints, the initial weight vectors for each optimization objective are calculated using the analytic hierarchy process (AHP): Construct a judgment matrix to determine the relative importance of each optimization objective based on resource priority weights. For example, in AI large-scale model training scenarios (high resource priority weights), the weight of the service quality objective > the weight of the resource utilization objective > the weight of the energy consumption objective; in cold data storage scenarios (low resource priority weights), the weight of the resource utilization objective > the weight of the energy consumption objective > the weight of the service quality objective. Perform a consistency check by calculating the consistency ratio (CR) of the judgment matrix, ensuring that CR is less than 0.1. If this is not met, adjust the elements of the judgment matrix until the consistency requirement is met. Calculate the weights by solving for the eigenvector corresponding to the largest eigenvalue of the judgment matrix using the eigenvalue method, and normalize it to obtain the initial weight vector. For example, the initial weight vector for the AI ​​scenario is service quality 0.45, resource utilization 0.35, and energy consumption 0.2; for the cold data scenario, it is resource utilization 0.4, energy consumption 0.35, and service quality 0.25, with a total element sum of 1.

[0106] Step 4.3: Fuse the initial weight vector with real-time status information from compute nodes, storage nodes, and network devices in the distributed storage system to define a state space. This state space includes at least the resource utilization, I / O load, network bandwidth occupancy, and physical resource type status of each node, specifically including: For computing nodes, real-time CPU utilization (by core), GPU utilization (computing cores and video memory), memory utilization, and NUMA node load distribution are recorded. For storage nodes, real-time SSD / HDD IOPS, throughput, average response time, remaining capacity, and write amplification factor are recorded. For network devices, real-time bandwidth utilization (uplink / downlink), data transmission latency, packet loss rate, and port connection status are recorded. The data collection frequency is consistent with step 1 (1 second / time for AI scenarios, 5 seconds / time for office scenarios). The collected data is standardized (mapped to the 0-1 range) to ensure uniform numerical scale. The initial weight vector is fused with the real-time state information to define a state space. The dimensions of the state space are determined by the number of real-time state indicators and the dimensions of the initial weight vector, specifically including: In terms of resource utilization, the utilization rates are: CPU utilization, GPU utilization, and memory utilization of compute nodes; SSD / HDD utilization of storage nodes; and network bandwidth utilization. In terms of load, the utilization rates are: compute node task load, storage node I / O load (read / write loads are counted separately), and network device data transmission load. In terms of physical resource status, the utilization rates are: storage device type (SSD / HDD identifier), GPU memory capacity level, and network device bandwidth level. In terms of weight, the initial weights are: resource utilization initial weight, service quality initial weight, and energy consumption initial weight.

[0107] Step 4.4: Based on the multi-objective decision-making hierarchy model and physical resource type constraints, define the action space. The action space includes data migration strategies, replica adjustment strategies, and resource allocation strategies based on physical resource types. Specifically, it includes defining the action space according to three main categories: data migration, replica adjustment, and resource allocation, based on the multi-objective decision-making hierarchy model and physical resource type constraints. Furthermore, the actions must match the physical resource types. Data migration strategies are categorized based on storage device type, such as migrating hot data from HDD to NVMeSSD (to meet the high I / O requirements of AI and OLTP scenarios), migrating cold data from SSD to HDD (to meet the cost control requirements of cold data scenarios), and balancing data migration between similar devices (to meet the load balancing requirements of office scenarios). During migration, the physical resource identifiers of the source node / target node must be determined (e.g., R03-S08-SSD01 to R03-S08-SSD02).

[0108] The replica adjustment strategy, combined with the physical resource redundancy setting, such as increasing the number of replicas of frequently accessed data on NVMeSSDs (from 2 replicas to 3 replicas) to improve the data reliability of AI scenarios, reducing the number of replicas of cold data on HDDs (from 3 replicas to 2 replicas) to reduce the storage occupation of cold data scenarios, and placing replicas across racks to improve disaster recovery capabilities in OLTP scenarios; when adjusting, it is necessary to ensure that the number of replicas is not lower than the minimum redundancy requirement of the system (such as a minimum of 2 replicas).

[0109] Resource allocation strategies are refined according to physical resource types. For example, GPU nodes with 24GB or more of video memory are allocated to AI tasks, high IOPS SSDs (3000MB / s or more) are allocated to OLTP tasks, ordinary CPU nodes (4 cores or more) are allocated to office tasks, and large-capacity HDDs (10TB or more) are allocated to cold data tasks. Real-time resource status should be taken into account during allocation to avoid resource over-provisioning.

[0110] Each strategy is transformed into quantifiable action parameters. For example, data migration strategies include the amount of data to be migrated (GB) and the upper limit of the migration rate (MB / s); replica adjustment strategies include the number of replicas after adjustment and the number of nodes distributed across the replicas; resource allocation strategies include the number of CPU cores, the size of GPU memory, and the storage capacity. At the same time, physical resource constraints are added to the actions, such as the NVMeSSD data migration rate not exceeding 80% of the device's maximum throughput and the storage utilization of a single node after replica adjustment not exceeding 90%, to ensure that the actions are executable and do not compromise system stability.

[0111] Step 4.5: Determine the reward function based on the optimization objective and physical resource utilization efficiency. This reward function comprehensively considers the balance of resource allocation, service quality satisfaction, energy consumption indicators, and physical resource suitability. Specifically, it includes four dimensions: the balance of resource allocation, service quality satisfaction, energy consumption indicators, and physical resource suitability. Each dimension corresponds to a quantitative indicator and weight. Resource allocation balance is measured using the Gini coefficient to assess the difference in utilization of the same type of resources across nodes; a smaller Gini coefficient results in a higher reward. The weight is set according to the resource utilization weight in the initial weight vector (e.g., 0.35 for the AI ​​scenario). Service quality satisfaction is assessed by comparing actual service quality indicators (e.g., I / O response time, access success rate) with scenario requirement indicators in the mapping table; positive rewards are given for meeting requirements, and negative rewards for not meeting them. The weight is set according to the initial service quality weight (e.g., 0.45 for the AI ​​scenario). Energy consumption is assessed by calculating the difference between the total system energy consumption (the sum of energy consumption across all nodes) and the optimal energy consumption (theoretical energy consumption after low-load nodes go into sleep mode); a smaller difference results in a higher reward. The weight is set according to the initial energy consumption weight (e.g., 0.2 for the AI ​​scenario). Physical resource adaptability is assessed by judging the degree of matching between resource scheduling actions and physical resource types (e.g., whether an AI task is allocated an NVMe SSD); a perfect match results in full marks, a partial match results in partial marks, and a mismatch results in negative marks. The weight is a fixed value of 0.2 (to ensure action and hardware compatibility), and the sum of the weights for the four dimensions is 1.2 (for adapting to multi-objective collaborative optimization).

[0112] The reward function is calculated using a weighted summation method, and the formula logic is as follows: Total reward = (1 - Gini coefficient) × resource balance weight + (actual QoS / required QoS) × service quality weight + (1 - energy consumption difference / optimal energy consumption) × energy consumption weight + adaptation score × adaptation weight; where, if the actual QoS is lower than 80% of the required QoS, (actual QoS / required QoS) is calculated as 0 and the reward is deducted additionally; if the physical resource adaptation score is lower than 0.5, the total reward needs to be multiplied by a penalty coefficient of 0.8 to ensure that the action prioritizes adaptation to physical resource requirements; the reward value range is set from -10 to 10, with positive numbers indicating good action optimization effects and negative numbers indicating that the action causes a decrease in system performance; required QoS, or required service quality, refers to the service quality benchmark determined by the distributed storage system in the resource requirement priority mapping table for different business scenarios, which ensures the normal operation of the business and meets user expectations; actual QoS, or actual service quality, refers to the service quality level actually obtained by the current business, collected by the real-time monitoring module after the distributed storage system executes resource scheduling actions, and is a real-time feedback on the resource scheduling effect.

[0113] Wherein, the required QoS = (response time required QoS normalized value × response time weight) + (I / O performance required QoS normalized value × I / O performance weight) + (reliability required QoS normalized value × reliability weight) + (bandwidth required QoS normalized value × bandwidth weight).

[0114] QoS Calculation and Standardization for Response Time Requirements: Based on the determined physical resource type, obtain the default response time benchmark of the storage device and the default transmission delay benchmark of the corresponding network device, and calculate the physical resource benchmark value (the sum of the response time benchmark and the transmission delay benchmark); determine the satisfaction level adjustment coefficient, and set the differentiation coefficient according to the satisfaction level (very low, low, medium, high, very high), with the coefficient being larger for lower satisfaction (more urgent resource demand); multiply the physical resource benchmark value by the satisfaction level adjustment coefficient to obtain the original response time requirement QoS (unit: milliseconds); adopt reverse normalization (the smaller the response time, the better), and map the original value to the 0-1 range based on the maximum allowed response time and the minimum achievable response time of the system, to obtain the standardized value of the response time requirement QoS.

[0115] I / O performance requirements QoS calculation and standardization: Determine the baseline performance for the resource type, i.e., extract the default I / O performance baseline (such as random read / write IOPS or throughput, selecting core indicators according to scenario requirements) based on the storage device type; set the service load coefficient, i.e., set the coefficient according to the service load characteristics of the scenario (high load, medium load, low load), with the coefficient increasing as the load increases; calculate the original I / O performance requirement QoS, i.e., multiply the resource baseline performance by the service load coefficient, and then multiply by the satisfaction level adjustment coefficient to obtain the original I / O performance requirement QoS (unit: IOPS or MB / s, determined according to the selected indicator); standardize the process, i.e., use forward normalization (the higher the I / O performance, the better), using the maximum achievable I / O performance of the corresponding system resources as the baseline, map the original value to the 0-1 range to obtain the standardized value of the I / O performance requirement QoS.

[0116] Reliability requirement QoS calculation and standardization: Set a basic reliability baseline, which means determining the system's default reliability baseline value (such as data access success rate) and adjusting the baseline according to the preset number of replicas (the more replicas, the higher the baseline); add a scenario importance coefficient, which means setting a coefficient according to the importance of the business scenario (high importance, medium importance, low importance), with the coefficient being larger for higher importance; calculate the original reliability requirement QoS, which means multiplying the adjusted reliability baseline by the scenario importance coefficient, and taking 100% as the upper limit if the result exceeds 100%, to obtain the original reliability requirement QoS (unit: %); standardize the data, which means using forward normalization, mapping the original value to the 0-1 range with 100% as the baseline, to obtain the standardized reliability requirement QoS value.

[0117] Bandwidth requirement QoS calculation and standardization: Estimate the business data volume, i.e., statistically analyze the average data transmission volume per unit time in the target scenario to determine the basic bandwidth requirements of the business; set time conversion and redundancy coefficients, i.e., convert the data volume per unit time to the data volume per second, add redundancy coefficients (to cope with sudden traffic, the greater the business fluctuation, the larger the redundancy coefficient), and calculate the original bandwidth requirements; unify units and determine original values: unify the unit to Mbps (through the conversion rules between MB / s and Mbps) to obtain the original bandwidth requirement QoS (unit: Mbps); standardize the processing, i.e., use forward normalization (the larger the bandwidth, the better), using the maximum bandwidth that the corresponding network can provide as the benchmark, map the original value to the 0-1 range to obtain the standardized value of the bandwidth requirement QoS.

[0118] Wherein, actual QoS = (actual QoS normalized value for response time × response time weight) + (actual QoS normalized value for I / O performance × I / O performance weight) + (actual QoS normalized value for reliability × reliability weight) + (actual QoS normalized value for bandwidth × bandwidth weight) Actual QoS Calculation and Standardization of Response Time: Real-time data acquisition involves collecting real-time response time data of the target storage device and real-time transmission latency data of the corresponding network device at a frequency set according to the scenario, continuously for a preset duration; data preprocessing involves using preset rules (such as the IQR method) to remove outliers and invalid data, retaining valid data samples; calculating the actual QoS of the original response time involves averaging the valid data samples to obtain the average response time and average transmission latency, and adding the two together gives the actual QoS of the original response time (unit: milliseconds); standardization processing involves using reverse normalization logic consistent with the required QoS to map the original values ​​to the 0-1 range to obtain the standardized value of the actual QoS of the response time.

[0119] Actual QoS Calculation and Standardization of I / O Performance: Data collection is categorized into read and write operations, and real-time I / O performance data of the target storage device is collected at a set frequency (consistent with the selected indicators of the required QoS, such as IOPS or throughput), continuously for a preset duration; data filtering: invalid data caused by device offline or failure is filtered out, and valid samples are retained; the actual QoS of the original I / O performance is calculated, that is, the average performance of read and write operations is calculated for the valid samples, and the sum of the two is the actual QoS of the original I / O performance (unit: IOPS or MB / s); standardization processing, that is, using the same forward normalization logic as the required QoS, the original values ​​are mapped to the 0-1 range to obtain the standardized value of the actual QoS of I / O performance.

[0120] Reliability-based QoS calculation and standardization: Access record collection involves recording all data access requests within a preset time period for the target scenario, distinguishing between successful and failed accesses; data processing involves excluding failed records due to non-service quality factors (such as user operation errors) and retaining valid access records; calculating the original reliability actual QoS involves dividing the number of valid successful accesses by the total number of valid accesses and then multiplying by 100% to obtain the original reliability actual QoS (unit: %); and standardization processing involves using forward normalization logic consistent with the required QoS to map the original values ​​to the 0-1 range to obtain the standardized value of the reliability actual QoS.

[0121] Actual QoS calculation and standardization of bandwidth: Real-time bandwidth acquisition involves collecting uplink and downlink real-time bandwidth usage data of the target network port at a set frequency for a preset duration. Data processing involves removing invalid data caused by port failures or no data transmission, retaining valid bandwidth data samples. Calculating the actual QoS of the original bandwidth involves extracting the bandwidth peak value (uplink or downlink peak, taking the larger value) from the valid samples and converting it to a unified unit of Mbps to obtain the actual QoS of the original bandwidth (unit: Mbps). Standardization processing involves using forward normalization logic consistent with the required QoS to map the original value to the 0-1 range to obtain the standardized value of the actual QoS of the bandwidth.

[0122] Step 4.6: Based on the state space, action space, and reward function, construct a multi-objective optimization model. Specifically, this includes selecting the PPO algorithm as the basic framework of the multi-objective optimization model because the PPO algorithm is more stable in continuous action spaces and multi-objective optimization scenarios, and can avoid excessive parameter update amplitude through the clippedsurrogate objective function.

[0123] Integrate the state space defined in step 4.3, the action space defined in step 4.4, and the reward function determined in step 4.5 into the PPO framework: The network structure employs a fully connected neural network with two hidden layers as the policy network and the value network. The input layer dimension matches the state space dimension (e.g., 16-dimensional), the number of hidden layer nodes is set to 128 (balancing accuracy and efficiency), and the output layer dimension matches the action space dimension (e.g., 8 actions correspond to 8-dimensional output). The training parameters include a learning rate of 3×10⁻⁶. -4 The discount factor γ is set to 0.95 (emphasizing long-term rewards), the parameter λ is set to 0.9 (balancing bias and variance), the batch size is set to 64, and the number of iterations is set to 10, with network parameters updated 10 times per iteration. Constraint integration is performed, using physical resource type constraints (such as NVMeSSD required for AI tasks) as hard constraints for action selection. These constraints are filtered before the policy network outputs actions, retaining only actions that meet the constraints to ensure that the model outputs executable actions. After integration, the multi-objective optimization model can output scheduling actions (action space) that meet the constraints based on the real-time state of the system (state space), and feedback the action effects through the reward function, achieving multi-objective collaborative optimization of resource utilization, service quality, and energy consumption.

[0124] In a preferred embodiment of the present invention, step 5, predicting the resource status and load of each node in the distributed storage system based on a multi-objective optimization model, includes: Step 5.1: Real-time collection of performance metrics from each computing node, storage node, and network device in the distributed storage system. These performance metrics include at least CPU and GPU utilization from computing nodes, I / O throughput and read / write latency from SSDs and HDDs from storage nodes, and bandwidth utilization and transmission latency from network devices. The performance metrics are then normalized to obtain normalized performance metrics data. Specifically, this includes: real-time collection of CPU utilization (statistically by core dimension, including single-core utilization and overall average utilization), GPU utilization (computing core utilization and video memory utilization), with a collection frequency of 1 second / time for AI large-scale model training scenarios, 2 seconds / time for database OLTP scenarios, and 5 seconds / time for office applications and cold data storage scenarios; and collection of I / O throughput (read throughput and write throughput are counted separately) and read / write latency (average latency and 99th percentile latency) from SSDs and HDDs, covering all scenarios. The scene acquisition frequency is uniformly set to 2 seconds / time, and the physical partition identifier of the storage device is recorded synchronously. The bandwidth utilization (uplink bandwidth utilization, downlink bandwidth utilization) and transmission latency (end-to-end latency, inter-port latency) of each port are collected, and the acquisition frequency is consistent with that of the computing nodes. At the same time, the source node and destination node identifiers of the data transmission are recorded. The IQR outlier detection method is used to replace outliers with the median of the indicator within the past 10 minutes. For the preprocessed performance indicators, the corresponding normalization method is used according to the indicator type. For proportional indicators such as CPU / GPU utilization and bandwidth utilization, the maximum and minimum values ​​are normalized to the 0-1 range. For absolute numerical indicators such as I / O throughput, read / write latency, and transmission latency, the maximum carrying capacity of the system hardware is used as the benchmark (such as the maximum I / O throughput of SSD and the maximum bandwidth of network devices). The original value is mapped to the 0-1 range by the maximum carrying capacity to ensure that the scale of all normalized performance indicator data is consistent.

[0125] Step 5.2 aligns and semantically fills the normalized performance metric data with the state space to construct the state vector for the current moment. Specifically, this involves matching the normalized performance metric data one by one according to the state space dimensions, based on the definition of the state space (including dimensions such as resource utilization, I / O load, network bandwidth utilization, and physical resource type status). First, align the resource utilization dimension by filling the resource utilization dimension of the state space with the CPU and GPU utilization of computing nodes, the SSD / HDD utilization of storage nodes (converted by the ratio of I / O throughput to maximum throughput), and the bandwidth utilization of network devices. Then, align the load and physical state dimensions by filling the load and physical resource state dimensions of the state space with the I / O load of storage nodes (read load and write load, converted by read / write throughput), the data transmission load of network devices (converted by bandwidth usage), and the storage device type identifier (SSD set to 1, HDD set to 0) and GPU memory level identifier. Ensure that each normalized performance metric has a corresponding definite state space dimension, with no missing or mismatched dimensions. For dimensions in the state space not directly covered by performance metrics (such as resource utilization weights, service quality weights, and energy consumption weights in the initial weight vector), read the corresponding values ​​from the configuration database of the multi-objective optimization model and fill them in. Retrieve the initial weight vector calculated in step 4.2 according to the current business scenario identifier (such as S-AI, S-DB-OLTP), and fill the weight dimension of the state space with the resource utilization weight, service quality weight, and energy consumption weight; after filling, a current state vector with dimensions completely consistent with the state space is formed, and each value in the vector is a normalized valid data with no null or invalid values.

[0126] Step 5.3: Based on the current state vector and the multi-objective optimization model, obtain the corresponding preliminary action decision through forward inference. Specifically, this includes converting the current state vector into an input format adapted to the policy network of the reinforcement learning model (PPO algorithm). If the state vector is N-dimensional, it is converted into a 1×N two-dimensional tensor with the data type set to floating-point to ensure matching with the dimension of the policy network input layer. Based on the latest training parameters (including weight matrix and bias terms) of the policy network in the multi-objective optimization model constructed in step 4.6, it is ensured that the inference uses the validated optimal model parameters. The input layer receives the state vector tensor and passes it to the double hidden layer (128 nodes per layer). The ReLU activation function is used to process and extract key features from the state vector (such as high CPU utilization, low I / O load, etc.). The output layer outputs the probability distribution of each action according to the action space dimension (including three types of actions: data migration, copy adjustment, and resource allocation). The top 5 actions with the highest probabilities are selected as preliminary action decisions. Each preliminary action decision must include specific parameters (such as determining the source node, target node, and amount of data to be migrated for data migration actions; determining the number of CPU cores and GPU memory size for resource allocation actions) to ensure that the actions are executable and verifiable.

[0127] Step 5.4 involves performing physical resource feasibility verification and strategy conflict detection on the preliminary action decisions to obtain a valid action set. Specifically, this includes verifying the hardware capacity of each preliminary action decision. For data migration, check if the remaining storage space on the target node is greater than the amount of data to be migrated, whether the target storage device type meets the requirements of the business scenario (e.g., the target node in an AI scenario must be an NVMe SSD), and whether the network bandwidth of the migration path meets the migration rate requirements. For replica adjustment, check if the number of replicas after adjustment meets the minimum redundancy requirements of the system (e.g., a minimum of 2 replicas), whether the replica distribution nodes are across racks (required for OLTP scenarios), and whether the storage nodes of the newly added replicas have sufficient I / O performance to support them. For resource allocation, check if the number of allocated CPU cores does not exceed the total number of cores on the node, whether the GPU memory does not exceed the total GPU memory on the node, and whether the storage capacity does not exceed the remaining capacity of the device. If any preliminary action decision does not meet any of the above conditions, it is determined to be infeasible and the action is removed.

[0128] For preliminary action decisions that have passed feasibility verification, further examine whether there are any strategy conflicts: Conflict types include resource over-allocation conflicts (such as multiple actions simultaneously allocating video memory to the same GPU node, resulting in a total exceeding the actual capacity) and action target conflicts (such as the same data being migrated to different nodes by two actions simultaneously). A conflict priority resolution mechanism is adopted, which sorts the actions according to their resource priority weights (from the mapping table in step 3.5), retaining high-priority actions and removing low-priority conflicting actions. If the priorities are the same, the actions that significantly improve service quality are retained (judged by the estimated value of the reward function corresponding to the action). Finally, the actions that pass the feasibility verification and conflict detection are aggregated to form an effective action set, ensuring that the actions in the set are conflict-free and executable.

[0129] Step 5.5: Based on the effective action set, simulate the execution of each resource scheduling action to predict the resource status and load of each node in the future. The resource status and load include at least the CPU and GPU utilization of each computing node, the I / O load of the SSD and HDD of each storage node, and the bandwidth utilization of network devices. Specifically, this includes: building a lightweight simulation execution environment based on the hardware topology (rack, server, and device hierarchy) and current resource status of the distributed storage system. The environment includes the hardware parameters of each node (such as the number of CPU cores, GPU memory, storage device capacity and maximum I / O performance, and network bandwidth limit), as well as the execution rules for resource scheduling actions (such as data migration rate = network bandwidth × 0.8, replica creation time = data volume ÷ storage device write rate); based on all actions in the effective action set, the simulation execution order is set according to the action priority order (resource priority weight from high to low).

[0130] The system simulates and executes each action sequentially. When performing data migration, it updates the storage capacity and I / O load of the source and target nodes in real time. When performing replica adjustment, it updates the replica count and I / O load of the storage nodes. When performing resource allocation, it updates the CPU / GPU utilization of the compute nodes and the storage usage of the storage nodes. It predicts the status and load over a future period, generating prediction data every minute according to a preset prediction period (e.g., 5 minutes, 10 minutes, matching the time window of the business scenario). This includes the predicted CPU / GPU utilization of each compute node (calculated based on the load change trend after the action), the predicted I / O load of each storage node's SSD / HDD (read load, write load, combined with historical load fluctuation patterns), and the predicted network device bandwidth utilization (calculated based on the transmission traffic generated by the action). After the prediction is completed, it outputs resource status and load prediction reports for each future time slice, and identifies and marks resource bottleneck warnings for each node (e.g., the predicted utilization of a certain CPU node will exceed 90%).

[0131] In this embodiment of the invention, the scope and frequency of performance indicator collection are refined according to node type to ensure real-time and comprehensive coverage of core indicators such as CPU / GPU utilization, I / O throughput, and bandwidth utilization. Simultaneously, outlier preprocessing and targeted normalization (differentiated processing for proportional and absolute numerical indicators) eliminate data noise and scale differences. Alignment of the state vector with the state space dimensions ensures that the normalized performance indicators accurately match the input requirements of the reinforcement learning model. Combined with semantic padding to complete key dimensions such as weights, a complete current state vector is formed. This construction method achieves seamless integration between the real-time state of the distributed storage system and the multi-objective optimization model, enabling the model to accurately perceive the current operating state of the system. Based on the current state vector, a high-probability preliminary action decision is output through forward inference of the reinforcement learning policy network, with specific execution parameters attached to the decision. This process relies on the training results of the multi-objective optimization model and can generate scheduling directions that conform to resource optimization objectives (resource utilization, service quality, energy balance) by combining the real-time state of the system (such as high CPU utilization and low I / O load), avoiding blind decision-making and improving the scientific and targeted nature of action decisions. The initial action decision is verified from two aspects: physical resource carrying capacity (such as target node storage space and device type matching) and policy conflicts (such as resource over-provisioning and action target conflicts). Infeasible and conflicting actions are eliminated to form an effective action set. This step can avoid scheduling failures caused by insufficient resources or policy contradictions in advance (such as insufficient space of data migration target node), ensure the feasibility of subsequent resource scheduling actions, and reduce system operation risks.

[0132] In a preferred embodiment of the present invention, step 6 involves performing lightweight rule scheduling based on the resource status and load of each node to obtain a preliminary scheduling scheme, including: Step 6.1: Based on the resource status and load of each node in the future, extract the predicted resource indicators of the current edge node and its adjacent computing and storage nodes. These predicted resource indicators include at least CPU and GPU utilization, SSD and HDD I / O load, and network bandwidth utilization. Specifically, this includes: defining an adjacent range with a radius of one rack unit around the edge computing node, and selecting adjacent computing nodes (other computing servers within the same rack) and adjacent storage nodes (servers containing storage devices directly connected to the edge node) within this range. This ensures that the extracted nodes have a direct hardware connection or short-path network association with the edge node, reducing cross-regional scheduling latency. From the future resource status and load prediction report output in Step 5.5, extract core predicted indicators by node type: For edge nodes and adjacent compute nodes, extract predicted CPU utilization (overall average utilization, highest single-core utilization) and GPU utilization (computation core utilization, video memory utilization) for two time slices in the next 5 and 10 minutes. Simultaneously record basic hardware parameters such as the total number of CPU cores and the total amount of GPU video memory. For adjacent storage nodes, extract predicted I / O load values ​​(read load percentage, write load percentage, total I / O throughput) for SSDs and HDDs in the next 5 and 10 minutes, and synchronously associate the physical partition identifiers of the storage devices and their corresponding remaining storage space. Associate network devices, extract predicted bandwidth utilization (uplink utilization, downlink utilization) for the network links between edge nodes and adjacent nodes in the next 5 and 10 minutes, as well as auxiliary indicators such as maximum link bandwidth and current transmission latency, ensuring that the indicators cover all dimensions of computing, storage, and network required for scheduling.

[0133] Step 6.2: Construct an edge node load assessment model based on computing resource utilization, storage I / O load, and network bandwidth utilization. Specifically, this includes dynamically assigning weights to the criteria according to the service characteristics of the edge nodes, with a total weight of 1. For compute-intensive edge services (such as AI inference), the weighting of the compute resource utilization criterion is 0.4, the weighting of the storage I / O load criterion is 0.2, and the weighting of the network bandwidth utilization criterion is 0.4, prioritizing compute and network resources. For storage-intensive edge services (such as edge caching), the weighting of the compute resource utilization criterion is 0.2, the weighting of the storage I / O load criterion is 0.5, and the weighting of the network bandwidth utilization criterion is 0.3, focusing on storage and network adaptation. For general-purpose edge services (such as office data processing), the weighting of the compute resource utilization criterion is 0.3, the weighting of the storage I / O load criterion is 0.3, and the weighting of the network bandwidth utilization criterion is 0.4, providing balanced coverage for the three types of resources.

[0134] The predictive resource indicators corresponding to the three criteria are quantified: The computing resource utilization criterion uses (predicted CPU utilization × 0.6 + predicted GPU utilization × 0.4) to calculate the overall utilization rate, with a higher weight for GPU utilization (for edge nodes with GPUs). If a node does not have a GPU, only CPU utilization is used for calculation. The storage I / O load criterion uses (predicted SSD I / O load × 0.7 + predicted HDD I / O load × 0.3) to calculate the overall I / O load, highlighting the impact of SSDs on high real-time services, while also considering the remaining space ratio of storage devices (when the remaining space ratio is less than 20%, the I / O load weight is increased by an additional 0.1). The network bandwidth utilization criterion uses (predicted uplink bandwidth utilization × 0.5 + predicted downlink bandwidth utilization × 0.5) to calculate the overall bandwidth utilization rate. If the predicted link transmission latency is higher than 5ms, the bandwidth utilization weight is reduced by 0.1 to avoid high-latency links affecting scheduling performance. The three types of quantitative results are integrated into a unified evaluation framework through weighted summation to form an edge node load evaluation model.

[0135] Step 6.3: Based on the load assessment model, calculate the comprehensive load score of the current edge node and each adjacent node, and determine the load level of the node based on the comprehensive load score and physical resource type constraints. Specifically, this includes: mapping the quantification results of each criterion to the range of 0-1 using the maximum and minimum value normalization method, where the calculated resource utilization rate, storage I / O load, and network bandwidth utilization rate are all positive indicators that the higher the value, the heavier the load; multiplying and summing the standardized criterion results according to the criterion weights determined in Step 6.2 to obtain the comprehensive load score of the current edge node and each adjacent node, with a score range of 0-1, where the higher the score, the heavier the node load; dividing the load level, with a preset 4-level load level standard: 0-0.3 is low load, 0.3-0.6 is medium load, 0.6-0.8 is high load, and 0.8-1.0 is overload; determining the initial load level of each node based on the comprehensive load score; and correcting the initial load level, if the storage node has SSD... If the remaining space is less than 10%, even if the overall load score is 0.5 (medium load), it will be upgraded to high load; if the number of CPU cores of the compute node is ≥32 and the overall load score is 0.7 (high load), it will be downgraded to medium load. This ensures that the load level matches the actual carrying capacity of the hardware and avoids misjudgment due to relying solely on the score.

[0136] Step 6.4: Based on the pre-configured lightweight scheduling rule base and the node load level, local scheduling instructions for data migration, replica placement, and task allocation are obtained. Specifically, the rule base is stored according to load level combinations and scheduling operation types. Each rule includes trigger conditions, operation parameters, and constraints. Triggering conditions: Determine the load level combination of the edge node and adjacent nodes (e.g., edge node overload plus adjacent compute nodes low load); define the specific execution requirements of the scheduling operation (e.g., upper limit of data migration rate, limit on the number of replicas placed, and resource ratio of task allocation); limit the physical resource type corresponding to the operation (e.g., the target node for data migration must be an NVMe SSD, and task allocation must match the CPU architecture).

[0137] During scheduling, the rule base is matched in the order of edge node load level, adjacent node load level, and physical resource type to select the unique rule that matches the current scenario. If the edge node's storage is overloaded and the adjacent storage node is underloaded, an instruction is generated to migrate data from the high-load SSD partition of the edge node to the low-load SSD partition of the adjacent node, specifying the amount of data to be migrated (not exceeding 30% of the remaining space of the adjacent node), the migration rate (not exceeding 50% of the network link bandwidth), and the priority of the migrated data (sorted by access frequency from low to high). If the edge node's computing is overloaded and the adjacent computing node is underloaded, an instruction is generated to add high-frequency access data replicas from the edge node to the local storage of the adjacent computing node, specifying the number of replicas (1-2), the type of replica storage device (must be consistent with the source data device type), and the replica synchronization frequency (synchronize once every 30 seconds). If the edge node's CPU is overloaded and the adjacent computing node is underloaded, an instruction is generated to allocate some computing tasks from the edge node to the adjacent computing node, specifying the task allocation ratio (not exceeding the CPU of the adjacent node). The remaining resources are allocated as follows: 40% of the total resources; task type (only stateless computing tasks are allocated); communication protocol after task migration (maintaining consistency with the original task); each local scheduling instruction is accompanied by an execution priority (levels 1-5, with level 1 being the highest) to ensure that subsequent integration is sorted by priority.

[0138] Step 6.5: Integrate all local scheduling instructions to form a preliminary scheduling scheme for edge nodes. The preliminary scheduling scheme includes at least the sequence of operations to be executed, target node information, and the required physical resource type configuration. Specifically, it includes: if multiple instructions involve the same target node resources, calculate the total resource occupancy (e.g., the total CPU percentage of multiple tasks allocated to the same computing node). If it exceeds 60% of the node's remaining resources, remove low-priority instructions according to instruction priority to ensure that the target node resources are not over-occupied; merge scheduling operations of the same type (e.g., multiple data migration instructions all pointing to the same adjacent node are merged into a single batch migration) to reduce the number of operations and reduce system scheduling overhead. List all local scheduling instructions in the execution order, specifying the execution time window for each instruction (e.g., data migration in the next 1-3 minutes, task allocation in the next 3-5 minutes) and the execution timeout threshold (e.g., data migration timeout is set to 5 minutes); label the physical identifier of the target node corresponding to each instruction (rack, server, and device number), the remaining resources of the target node (before scheduling), and the estimated resource usage after scheduling, ensuring that node information is traceable; determine the physical resource type and parameters required for each instruction (e.g., data migration requires NVMe SSD, task allocation requires x86 architecture CPU, and network links require gigabit or higher bandwidth), and also indicate the validity period of resource usage (e.g., task allocation resources only occupy 2 hours).

[0139] like Figure 2 As shown, the distributed storage resource intelligent scheduling device includes: The acquisition module is used to acquire multimodal data from the distributed storage system; analyze the multimodal data to extract multimodal data features, including text features and time-series features, to obtain feature vectors for different modalities; construct a multidimensional data space based on the storage server rack location and storage device physical partitions, and construct a tree-like abstract data structure based on this multidimensional data space; divide the abstract data structure into multiple partition units with physical server nodes as the basic unit, divide the partition units according to the data distribution characteristics to form multiple sub-regions, map the data to the corresponding sub-regions, and obtain resource adjustment parameters based on the data characteristics of each sub-region; The concatenation module is used to concatenate feature vectors from different modalities to obtain multimodal feature vectors. The module is used to predict user satisfaction based on multimodal feature vectors and resource adjustment parameters; to establish a resource demand priority mapping table for different business scenarios based on user satisfaction; and to build a multi-objective optimization model based on the resource demand priority mapping table. The prediction module is used to predict the resource status and load of each node in the distributed storage system based on a multi-objective optimization model; and to perform lightweight rule scheduling based on the resource status and load of each node to obtain a preliminary scheduling scheme.

[0140] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent scheduling of distributed storage resources, characterized in that, The method includes: Collect multimodal data from a distributed storage system; analyze the multimodal data to extract multimodal data features, including text features and time-series features, to obtain feature vectors for different modalities; construct a multidimensional data space based on the storage server rack location and physical partitions of the storage devices, and construct a tree-like abstract data structure based on this multidimensional data space; divide the abstract data structure into multiple partition units with physical server nodes as the basic unit, divide the partition units according to the data distribution characteristics to form multiple sub-regions, map the data to the corresponding sub-regions, and obtain resource adjustment parameters based on the data characteristics of each sub-region; Feature vectors from different modalities are concatenated to obtain multimodal feature vectors; Based on multimodal feature vectors and resource adjustment parameters, predict user satisfaction; based on user satisfaction, establish a resource demand priority mapping table for different business scenarios; Construct a multi-objective optimization model based on the resource demand priority mapping table; Based on a multi-objective optimization model, the resource status and load of each node in a distributed storage system are predicted. Based on the resource status and load of each node, lightweight rule scheduling is performed to obtain a preliminary scheduling scheme.

2. The intelligent scheduling method for distributed storage resources according to claim 1, characterized in that, The multimodal data includes CPU and GPU utilization metrics from computing nodes, I / O performance data from SSDs and HDDs from storage nodes, and bandwidth and latency data from network devices.

3. The intelligent scheduling method for distributed storage resources according to claim 2, characterized in that, The process of determining resource adjustment parameters includes: Multimodal data collected from computing nodes, storage nodes, and network devices are preprocessed and organized according to their physical attributes and logical relationships. A multidimensional data space is constructed with the storage server rack location and storage device physical partitions as spatial references. Based on the multidimensional data space, a tree-like abstract data structure for expressing resource characteristics is constructed, with physical server nodes as root nodes and storage devices as leaf nodes. Based on the distribution density and business relevance of multimodal data in the multidimensional data space, the tree-like abstract data structure is divided into regions to form multiple partition units with physical server nodes as the basic units. Cluster analysis is performed on the data distribution within each partition unit to divide it into sub-regions, and each data instance is mapped to its respective sub-region based on its physical location and access characteristics. Extract the data access frequency, data size, and response time characteristics of each sub-region, and calculate the resource weight coefficient of the corresponding sub-region based on the data access frequency, data size, and response time characteristics to obtain the resource adjustment parameters.

4. The intelligent scheduling method for distributed storage resources according to claim 3, characterized in that, Based on the distribution density and business relevance of multimodal data in the multidimensional data space, the tree-like abstract data structure is divided into regions, forming multiple partition units with physical server nodes as the basic unit, including: Based on the hierarchical relationship between physical server nodes and storage devices in the tree-like abstract data structure, a radial dimension for measuring the distribution density of data on physical storage devices and an axial dimension for distinguishing different rack server node data types are defined, and a cylindrical coordinate mapping model is constructed. Based on the cylindrical coordinate mapping model, each data point in the multidimensional data space is mapped to the corresponding cylindrical coordinates according to its physical server node and storage device partition. The data is initially divided according to the axial dimension to form several candidate partitions based on physical server nodes. For each candidate partition, calculate the distribution density of its internal data points at the physical storage device level to obtain the adjusted partitions. The adjusted partitions are merged and optimized based on the business relevance between their data attributes and the topological proximity of physical server nodes to form multiple partition units with physical server nodes as the basic unit.

5. The intelligent scheduling method for distributed storage resources according to claim 4, characterized in that, Cluster analysis is performed on the data distribution within each partition unit to divide it into sub-regions, and each data instance is mapped to its respective sub-region based on its physical location and access characteristics, including: For each partition unit with physical server nodes as the basic unit, the spatial coordinates, physical location attributes, and access feature attributes of all data points within it are extracted to form a dataset to be clustered. Based on the physical location distribution characteristics of the data points in the dataset to be clustered, its initial convex hull is obtained; each edge of the initial convex hull is traversed, and the concavity depth of the data points in its neighborhood is calculated; according to the preset concavity depth threshold, the concave points that meet the conditions are identified and added to the boundary point set until no new concave points that meet the conditions are generated, so as to obtain a concave polygon constructed from the boundary point set. Using the concave polygon as the constraint boundary, cluster analysis is performed on the dataset to be clustered, and data points located inside the same concave polygon, physically adjacent and with similar access characteristics are grouped into the same sub-region. For each sub-region obtained by clustering, calculate the physical location center and access feature center of its data points, and map each data instance in the partition unit to its corresponding sub-region based on its physical location similarity and access feature similarity.

6. The intelligent scheduling method for distributed storage resources according to claim 5, characterized in that, Predict user satisfaction based on multimodal feature vectors and resource adjustment parameters; Based on user satisfaction, establish a resource requirement priority mapping table for different business scenarios, including: The multimodal feature vector is weighted and fused with the resource adjustment parameters to obtain the fused feature vector; Historical resource usage data and real-time monitoring data from compute nodes, storage nodes and network devices in a distributed storage system are acquired and standardized to extract time-series and statistical features related to the distribution of physical resources. The fused feature vector is concatenated with temporal features and statistical features to form an input feature set; Based on the input feature set, a prediction model is constructed; based on the prediction model, the predicted value of user satisfaction is obtained. Based on the predicted user satisfaction value, and combined with the preset business scenario classification tags and physical resource constraints, a resource demand priority mapping table is established for different business scenarios. The mapping table includes at least the scenario identifier, satisfaction level, corresponding resource priority weight, and required physical resource type.

7. The intelligent scheduling method for distributed storage resources according to claim 6, characterized in that, Based on the resource demand priority mapping table, construct a multi-objective optimization model, including: Parse the resource requirement priority mapping table to obtain the resource priority weights and the required physical resource types under different business scenarios; A multi-objective decision-making hierarchical structure model is constructed with resource utilization, service quality and energy consumption as optimization objectives, and the initial weight vector of each optimization objective is calculated based on the resource priority weight and physical resource type constraints. The initial weight vector is fused with real-time status information from computing nodes, storage nodes and network devices in the distributed storage system to define a state space. The state space includes at least the resource utilization, I / O load, network bandwidth utilization and physical resource type status of each node. Based on the multi-objective decision-making hierarchical structure model and physical resource type constraints, an action space is defined, which includes data migration strategy, replica adjustment strategy and resource allocation strategy based on physical resource type. A reward function is determined based on the optimization objective and the efficiency of physical resource utilization. The reward function comprehensively considers the balance of resource allocation, service quality satisfaction, energy consumption indicators and physical resource suitability. Based on the state space, action space, and reward function, a multi-objective optimization model is constructed.

8. The intelligent scheduling method for distributed storage resources according to claim 7, characterized in that, Based on a multi-objective optimization model, the resource status and load of each node in a distributed storage system are predicted, including: The system collects performance metrics of each computing node, storage node, and network device in the distributed storage system in real time. These performance metrics include at least the CPU and GPU utilization from computing nodes, the I / O throughput and read / write latency from SSDs and HDDs from storage nodes, and the bandwidth utilization and transmission latency from network devices. The performance metrics are then normalized to obtain normalized performance metric data. The normalized performance metrics data are aligned with the state space and semantically filled to construct the state vector at the current moment; Based on the current state vector and the multi-objective optimization model, the corresponding preliminary action decision is obtained through forward reasoning; The preliminary action decisions are then subjected to physical resource feasibility verification and strategy conflict detection to obtain an effective action set; Based on the set of effective actions, simulate the execution of each resource scheduling action to predict the resource status and load of each node in the future. The resource status and load include at least the CPU and GPU utilization of each computing node, the I / O load of SSD and HDD of each storage node, and the bandwidth utilization of network devices.

9. The intelligent scheduling method for distributed storage resources according to claim 8, characterized in that, Based on the resource status and load of each node, lightweight rule-based scheduling is performed to obtain a preliminary scheduling scheme, including: Based on the resource status and load of each node in the future, the predicted resource indicators of the current edge node and its adjacent computing and storage nodes are extracted. The predicted resource indicators include at least CPU and GPU utilization, SSD and HDD I / O load and network bandwidth utilization. Construct an edge node load assessment model based on computing resource utilization, storage I / O load, and network bandwidth utilization. Based on the load assessment model, calculate the comprehensive load score of the current edge node and each adjacent node, and determine the load level of the node based on the comprehensive load score and physical resource type constraints. Based on a pre-configured lightweight scheduling rule base and the load level of the nodes, local scheduling instructions are obtained for data migration, replica placement and task allocation. Integrate all local scheduling instructions to form a preliminary scheduling scheme for edge nodes. The preliminary scheduling scheme includes at least the sequence of operations to be executed, target node information, and the required physical resource type configuration.

10. A distributed storage resource intelligent scheduling device, which implements the method as described in any one of claims 1 to 9, characterized in that, include: The acquisition module is used to acquire multimodal data from a distributed storage system. The multimodal data is analyzed to extract multimodal data features, including textual and temporal features, to obtain feature vectors for different modalities. A multidimensional data space is constructed based on the storage server rack location and physical partitions of the storage devices. A tree-like abstract data structure is then constructed based on this multidimensional data space. In the abstract data structure, multiple partition units are divided using physical server nodes as the basic unit. The partition units are further divided according to the data distribution characteristics to form multiple sub-regions. Data is mapped to the corresponding sub-regions, and resource adjustment parameters are obtained based on the data characteristics of each sub-region. The concatenation module is used to concatenate feature vectors from different modalities to obtain multimodal feature vectors. The module is used to predict user satisfaction based on multimodal feature vectors and resource adjustment parameters; to establish a resource demand priority mapping table for different business scenarios based on user satisfaction; and to build a multi-objective optimization model based on the resource demand priority mapping table. The prediction module is used to predict the resource status and load of each node in the distributed storage system based on a multi-objective optimization model; and to perform lightweight rule scheduling based on the resource status and load of each node to obtain a preliminary scheduling scheme.

Images