Artificial intelligence-based storage resource dynamic allocation method and system

Abstract

The application provides an artificial intelligence-based storage resource dynamic allocation method and system, relates to the technical field of storage resource dynamic allocation, and obtains the real-time response delay, data transmission bandwidth and node computing power data of each storage node in the current stage of a target task, cooperatively processes the data, and obtains a storage resource real-time performance data packet containing the real-time load state of each storage node; based on an artificial intelligence model corresponding to the task stage, the data packet is analyzed to obtain a storage resource allocation target value; through a dynamic allocation optimization algorithm, a resource scheduling strategy is formulated and a real-time control signal is generated according to the target value, and the dynamic allocation and scheduling of storage resources among each storage node are executed accordingly, so that the artificial intelligence-based storage resource dynamic allocation method can realize the precise dynamic allocation and scheduling of storage resources that meet the requirements of a target task through data cooperative processing, AI analysis suitable for the task stage and a dynamic optimization algorithm.

Description

Technical Field

[0001] This application relates to the technical field of dynamic allocation of storage resources, and in particular to a method and system for dynamic allocation of storage resources based on artificial intelligence. Background Technology

[0002] In scenarios such as cloud computing, big data processing, and artificial intelligence training that rely on the collaboration of multiple storage nodes, the target task typically includes different stages such as data acquisition, computational processing, and result output. Each stage has significantly different requirements for the response speed, data transmission capacity, and computational support capabilities of the storage nodes. For example, the data acquisition stage requires storage nodes to have high data transmission bandwidth, while the computational processing stage requires high computing power. If storage resource allocation cannot be dynamically adjusted according to the task stage, it can easily lead to task execution lag or resource idleness. Therefore, there is an urgent need for a dynamic storage resource allocation solution that can adapt to the needs of each task stage.

[0003] Currently, most existing storage resource allocation technologies rely on preset resource allocation templates, such as allocating resources evenly based on the number of storage nodes or triggering resource adjustments based on a single real-time metric, such as the node's current bandwidth utilization or CPU utilization. Specifically, the process involves periodically collecting basic performance data from each storage node, then comparing it against preset load thresholds to determine if adjustments are needed. During the adjustment process, fixed algorithms such as round-robin or weighted averaging are often used to determine the resource allocation ratio among nodes, maintaining a roughly balanced load across all nodes.

[0004] However, existing technologies have a core drawback, the most significant being the failure to develop resource allocation strategies tailored to the specific stage characteristics of the target task. Instead, adjustments are made based solely on general templates or single metrics, leading to a disconnect between resource allocation and the actual needs of each task stage. For instance, when a task is in the data transmission stage, it requires high bandwidth support, but existing technologies may allocate computing resources according to an average ratio. Conversely, when a task enters the computation stage, it requires high computing power, yet existing technologies still maintain high bandwidth allocation, resulting in resource mismatch. This fails to achieve precise matching between resources and task requirements, thereby affecting task execution efficiency or causing resource waste. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for dynamic allocation of storage resources based on artificial intelligence, so as to solve the problem that storage resources and task requirements cannot be accurately matched in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for dynamically allocating storage resources based on artificial intelligence, comprising:

[0007] Obtain real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task;

[0008] The real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data are processed together to obtain a real-time performance data packet of storage resources that includes the real-time load status of each storage node.

[0009] Based on the artificial intelligence model corresponding to the task stage, the real-time performance data packet of the storage resources is analyzed to obtain the target value for storage resource allocation.

[0010] The dynamic allocation optimization algorithm is used to formulate a resource scheduling strategy based on the target value of the storage resource allocation, and generate a real-time control signal corresponding to the resource scheduling strategy. Based on the real-time control signal, the dynamic allocation and scheduling of storage resources among the storage nodes are executed.

[0011] Optionally, the step of formulating a resource scheduling strategy based on the storage resource allocation target value using a dynamic allocation optimization algorithm, and generating a real-time control signal corresponding to the resource scheduling strategy, includes:

[0012] Based on the storage resource allocation target value, and combined with the hardware specifications and task load characteristics of each storage node, calculate the resource requirement list of each storage node under the current task stage. The resource requirement list includes: storage capacity, read / write throughput, storage resource type and quantity.

[0013] Collect real-time available resource data from all storage nodes and compile them into a global available resource list. Compare the resource demand list of each storage node with the global available resource list item by item. Based on the comparison results, quantify and calculate the supply and demand difference of various resources. The part where demand exceeds supply is the resource gap, and the part where supply exceeds demand is the resource surplus.

[0014] Based on the supply and demand difference of the various resources, the resource allocation order of each storage node is determined through a dynamic allocation optimization algorithm.

[0015] According to the resource allocation order, the resource requirement list of each storage node is adjusted to form a resource scheduling strategy for each storage node. The resource scheduling strategy includes: resource allocation path, adjustment range and implementation time.

[0016] The resource scheduling strategy is transformed into rules to obtain instruction parameters. The instruction parameters are then encoded according to a preset signal format to generate a real-time control signal corresponding to the resource scheduling strategy.

[0017] Optionally, the step of determining the resource allocation order of each storage node based on the supply-demand difference of the various types of resources through a dynamic allocation optimization algorithm includes:

[0018] Based on the supply and demand gaps of various resources, the corresponding allocation gaps or surplus information is determined. The allocation gaps or surplus information includes: the type, scale, and scope of impact of the resource gaps or surpluses.

[0019] The allocation gap or surplus information is classified according to the type of response latency resources, data transmission bandwidth resources, and computing power resources to obtain the gap or surplus degree corresponding to each type of resource.

[0020] The shortage or surplus of each type of resource is compared with the corresponding preset resource importance level to determine the allocation priority of the corresponding resource in the current task stage.

[0021] Extract the required value of each type of resource in each storage node and the urgency level of the current task carried by each storage node from the resource requirement list;

[0022] The required value is associated with the urgency level identifier. Based on the association result, the storage nodes under the same resource are sorted according to the preset sorting rules to obtain the node allocation sub-order corresponding to the resource.

[0023] According to the allocation priority, all the node allocation sub-orders are concatenated and integrated to obtain the resource allocation order of each storage node.

[0024] Secondly, this application provides a dynamic storage resource allocation system based on artificial intelligence, comprising:

[0025] The acquisition module is used to acquire real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task.

[0026] The processing module is used to collaboratively process the real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node.

[0027] The analysis module is used to perform performance analysis on the real-time performance data packets of the storage resources based on the artificial intelligence model corresponding to the task stage, so as to obtain the target value for storage resource allocation.

[0028] The generation module is used to formulate a resource scheduling strategy based on the storage resource allocation target value through a dynamic allocation optimization algorithm, generate a real-time control signal corresponding to the resource scheduling strategy, and execute the dynamic allocation and scheduling of storage resources among storage nodes based on the real-time control signal.

[0029] Thirdly, this application provides an electronic device, comprising:

[0030] Memory, used to store computer programs;

[0031] A processor, configured to implement the steps of the AI-based dynamic allocation method for storage resources as described in the first aspect above when executing the computer program.

[0032] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the artificial intelligence-based dynamic allocation method for storage resources as described in the first aspect above.

[0033] The AI-based dynamic allocation method for storage resources provided in this application acquires real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task. It then collaboratively processes these data to obtain a real-time performance data packet containing the real-time load status of each storage node. Based on an AI model corresponding to the task stage, it performs performance analysis on the real-time performance data packet to obtain a target value for storage resource allocation. Finally, through a dynamic allocation optimization algorithm, it formulates a resource scheduling strategy based on the target value and generates a real-time control signal corresponding to the strategy. Based on this real-time control signal, it executes the dynamic allocation and scheduling of storage resources among the storage nodes.

[0034] The technical solution of this application has the following beneficial effects:

[0035] This application ensures that the collected data is strongly correlated with the current stage of the target task and the actual operating status of each storage node, avoiding interference from irrelevant data. This provides accurate and targeted basic data support for subsequent storage resource allocation, guaranteeing the effectiveness of input in subsequent processing stages. Through multi-data collaborative integration, it comprehensively reflects the performance and real-time load of each storage node, forming complete and coherent performance data results, providing comprehensive and reliable input information for subsequent artificial intelligence model analysis. Relying on an artificial intelligence model adapted to the task stage, the performance analysis process closely aligns with the specific needs of the current task stage, avoiding the problem of insufficient adaptability of general analysis. This ensures that the output storage resource allocation target value accurately matches the characteristics of the task stage and the actual performance of the nodes, providing a scientific and demand-oriented basis for subsequent resource scheduling. By transforming the abstract allocation target value into a specific executable resource scheduling strategy and then implementing it through real-time control signals, dynamic adaptation of resources among storage nodes is achieved. This ensures that resource allocation is synchronized with task requirements and node status in real time, effectively improving storage resource utilization efficiency and task execution adaptability.

[0036] Furthermore, this application calculates the resource requirement list for each node in the current task stage based on the target value of storage resource allocation and the hardware specifications and task load characteristics of each storage node; then, it collects real-time available resource data of all storage nodes and summarizes it to form a global available resource list; it compares the resource requirement list of each node with the global available resource list item by item, and quantifies the supply and demand difference of various resources based on the comparison results; then, based on the supply and demand difference of various resources, it determines the resource allocation order of each storage node through a dynamic allocation optimization algorithm; subsequently, it adjusts the resource requirement list of each node according to the allocation order to form a resource scheduling strategy for each storage node, including the resource allocation path, adjustment range, and implementation time; finally, it performs rule conversion on the resource scheduling strategy to obtain instruction parameters, encodes the instruction parameters according to a preset signal format, and generates a real-time control signal corresponding to the resource scheduling strategy.

[0037] This application ensures the quantification and clarity of resource requirements for each storage node by calculating a clear list of resource needs, avoiding allocation deviations caused by ambiguous requirements. It accurately grasps the supply and demand status of various resources within the system by collecting and summarizing globally available resources and comparing them to determine the supply-demand gap, providing objective data support for resource allocation. A dynamic algorithm determines the resource allocation order, ensuring an orderly and reasonable allocation process that prioritizes key nodes and core needs. Adjustments create a resource scheduling strategy with specific execution details, making resource allocation operable and avoiding vague strategies. Rule conversion and encoding generate real-time control signals, ensuring that the scheduling strategy can be effectively identified and executed by the storage resource management module. Ultimately, this achieves accurate, orderly, and efficient dynamic allocation of storage resources within the system, solving resource shortages while avoiding resource surplus waste, and improving resource utilization and task execution adaptability. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a method for dynamically allocating storage resources based on artificial intelligence, provided in an embodiment of this application;

[0040] Figure 2 A schematic diagram illustrating a specific implementation of an artificial intelligence-based dynamic allocation method for storage resources provided in this application embodiment;

[0041] Figure 3A schematic diagram of the structure of a storage resource dynamic allocation system based on artificial intelligence provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] In scenarios requiring multi-storage node collaboration, such as cloud computing, big data processing, and AI training, the target task typically goes through different stages, including data acquisition, computation, and result output. Each stage has significantly different requirements for storage nodes; for example, the data acquisition stage requires rapid data transmission, while the computation stage demands stronger computational support. However, most existing storage resource allocation technologies rely on pre-set fixed templates, such as allocating resources evenly based on the number of nodes or adjusting resources according to a single metric, such as the current transmission or computation occupancy rate of a node, without considering the specific characteristics of each task stage. This approach easily leads to resource misallocation. For instance, during the data acquisition stage, transmission resources should be prioritized, but a large amount of computational resources are allocated instead; conversely, during the computation stage, more computational resources are needed, but high transmission resource configurations are maintained. Ultimately, this either results in wasted idle resources or slows down task progress due to insufficient critical resources.

[0044] To address the disconnect between resource allocation and task-stage requirements in existing technologies, this application proposes an AI-based dynamic storage resource allocation method. The core process is as follows: First, acquire real-time response speed, data transmission speed, and computing power data of each storage node at the current stage of the target task; then, integrate and process this data to clarify the real-time load of each node; next, analyze this data using an AI model adapted to the current task stage to determine resource allocation targets that meet the stage requirements; finally, use a dynamic optimization algorithm to formulate a specific resource scheduling scheme and generate control signals to execute resource allocation between nodes. This scheme, through "task-stage adapted AI analysis," avoids the limitations of existing technologies that rely on general templates, ensuring that resource allocation aligns with the current stage requirements; through the integrated processing of multiple types of data, it overcomes the one-sidedness of judging by a single indicator, accurately grasping the actual state of nodes; ultimately, it achieves dual adaptation of resource allocation with task stage and node load, effectively solving the resource mismatch problem in existing technologies, improving resource utilization, and ensuring efficient task execution.

[0045] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] The core of this application is to provide a method for dynamic allocation of storage resources based on artificial intelligence, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0047] S101. Obtain the real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task.

[0048] In the above scheme, the target task refers to the business or technical task to be completed collaboratively by multiple storage nodes, such as data processing and model training; the task stage refers to the different processing links in the execution of the target task, such as data import, computation processing, and result export, and the performance requirements of storage nodes differ in different stages; the storage node refers to the hardware or logical unit used to carry storage resources and participate in the data storage and interaction of the target task, such as the independent storage server in the distributed storage system and the logical storage unit in the cloud storage; real-time response latency data refers to the real-time time data from receiving a data request to returning a response result, which is used to reflect the node's response speed; real-time data transmission bandwidth data refers to the amount of data transmission that the storage node can actually complete per unit time, which is used to reflect the node's data transmission capability; real-time node computing power data refers to the computing resource data that the storage node can currently use for data processing, which usually includes CPU idle rate, available memory, and number of idle computing units, which is used to reflect the node's computing support capability.

[0049] In this application example, firstly, the target task and its current task stage are determined: First, the specific type of the target task to be executed is clarified, such as a distributed AI model training task or a big data batch processing task. Then, based on the preset execution process or real-time progress of the target task, its current task stage is determined, and the core requirements of the current stage for the performance of storage nodes are clarified. Based on this result, all storage nodes participating in the current task stage are screened: According to the resource configuration list or system association records of the target task, storage nodes that have data interaction or resource support relationships with the current task stage are screened, and the target node list for data collection is determined to avoid interference from irrelevant node data.

[0050] Next, using the established list of target storage nodes, real-time response latency data for each node is collected: A "request-response" testing method is employed, sending a preset number of test data requests to each target storage node and recording the time from sending the request to receiving the response. Outliers are removed, and the average value is taken to obtain the real-time response latency data for each node. Based on the same list of target storage nodes, real-time data transmission bandwidth data for each node is collected: A "real-time traffic monitoring method" is used, recording the total amount of data received and sent by the target node within a preset time period using the system's built-in traffic monitoring tool. The real-time data transmission bandwidth data is then calculated according to the relationship "transmission bandwidth = total transmission data volume / monitoring time period." This data reflects the node's current transmission capacity and supplements the transmission dimension of performance analysis.

[0051] Finally, based on the target storage node list, real-time node computing power data of each node is collected: using the "resource occupancy monitoring method", the system resource monitoring tool is used to obtain the current CPU idle rate, memory availability and other core computing resource parameters of the target node. Then, combined with the preset computing power conversion rules, such as "computing power = total number of CPU cores × CPU idle rate + memory availability × preset computing power coefficient", the real-time node computing power data of each node is calculated. This data is used to reflect the current computing support capability of the node and improve the calculation dimension basis for performance analysis.

[0052] In a practical application, a technology company is executing a distributed AI model training task. This task is pre-defined as consisting of three stages: data import, model training, and result export. The task progress monitoring system determines that the current stage is data import. Based on the resource configuration list for this task, the storage nodes participating in the data import stage are selected as node A, node B, and node C.

[0053] First, when collecting real-time response latency data for each node, 10 training data read requests of 200MB each were sent to nodes A, B, and C respectively. After recording the time taken for each request and removing outliers, the average time taken for all requests after removing outliers was calculated to obtain a real-time response latency of 25 milliseconds for node A, 30 milliseconds for node B, and 22 milliseconds for node C. These data are used for subsequent analysis of the response speed adaptability of each node.

[0054] Next, when collecting real-time data transmission bandwidth data of each node, the transmission data of each node within 5 minutes is recorded by the traffic monitoring tool: Node A receives 20GB of data and sends 0.3GB of data, Node B receives 18GB of data and sends 0.2GB of data, and Node C receives 22GB of data and sends 0.4GB of data. The real-time data transmission bandwidth is calculated using the formula: Real-time data transmission bandwidth = (Received data volume + Sent data volume) ÷ Monitoring time, where the units for received and sent data volume are MB, and the unit for monitoring time is seconds. First, convert the data volume to MB (1GB = 1024MB), resulting in a total data transmission volume of 20 × 1024 + 0.3 × 1024 = 20787.2 MB for node A. With a monitoring time of 5 minutes = 300 seconds, the real-time data transmission bandwidth for node A is approximately 20787.2 ÷ 300 ≈ 69.3 MB / s. Similarly, the bandwidth for node B is approximately (18 × 1024 + 0.2 × 1024) ÷ 300 ≈ 62.1 MB / s, and for node C, it is approximately (22 × 1024 + 0.4 × 1024) ÷ 300 ≈ 75.5 MB / s. These data are used for subsequent analysis of the matching degree of transmission capabilities among the nodes.

[0055] Then, when collecting real-time node computing power data for each node, parameters are obtained through resource monitoring tools: Node A has a total of 32 CPU cores, a CPU idle rate of 50%, and 16GB of available memory; Node B has a total of 32 CPU cores, a CPU idle rate of 45%, and 12GB of available memory; Node C has a total of 32 CPU cores, a CPU idle rate of 55%, and 18GB of available memory. Real-time node computing power is calculated according to the formula: Real-time node computing power = Total CPU cores × CPU idle rate + Available memory × Memory computing power coefficient, where the memory computing power coefficient is preset to 0. 8. The real-time node computing power of node A is obtained as 32×50%+16×0.8=16+12.8=28.8; the real-time node computing power of node B is 32×45%+12×0.8=14.4+9.6=24; and the real-time node computing power of node C is 32×55%+18×0.8=17.6+14.4=32. All these real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data will be passed into the subsequent S102 step to generate a real-time performance data packet of storage resources containing the real-time load status of each storage node.

[0056] The aforementioned S101 overall solution, by first clearly defining the target task and its current stage, and then selectively filtering storage nodes, avoids collecting data from irrelevant nodes or data unrelated to the current stage, ensuring the relevance of data collection. Simultaneously, it uses standardized methods to collect three types of real-time data reflecting the storage node's response speed, transmission capacity, and computing support capabilities, comprehensively covering the core performance dimensions required for subsequent resource allocation analysis and avoiding the bias of a single data dimension. The three types of data ultimately obtained accurately reflect the actual operating status of each storage node at the current task stage, providing reliable and accurate basic input for collaborative data processing and AI model performance analysis in subsequent steps. This effectively ensures the accuracy of subsequent storage resource allocation analysis and lays the data foundation for achieving dynamic resource allocation tailored to task requirements.

[0057] S102. The real-time response delay data, the real-time data transmission bandwidth data, and the real-time node computing power data are processed together to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node.

[0058] Optionally, step S102 involves co-processing the real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node, including:

[0059] Step 1021: Based on the real-time response latency data, calculate the average latency and latency jitter coefficient within a unit time window to form a latency feature set. Step 1022: Perform sliding window statistics on the real-time data transmission bandwidth data to calculate bandwidth utilization, peak bandwidth percentage, bandwidth fluctuation frequency, and effective transmission rate, generating a bandwidth performance index set. Step 1023: Normalize the real-time node computing power data and construct a computing power load assessment value by combining node CPU utilization, memory occupancy, number of input / output operations per second, and task queue length. Step 1024: Utilize a three-dimensional collaborative analysis model to perform spatiotemporal correlation mapping on the latency feature set, the bandwidth performance index set, and the computing power load assessment value to generate a comprehensive load index for each storage node, and determine the load status level based on the comprehensive load index. Step 1025: Structure and integrate the latency feature set, the bandwidth performance index set, the computing power load assessment value, the comprehensive load index, and the load status level of each storage node to form a real-time performance data package for storage resources.

[0060] In the above scheme, the unit time window refers to a fixed time interval used for statistical latency data; the average latency refers to the arithmetic mean of all real-time response latency data within the unit time window; the latency jitter coefficient refers to the ratio of the standard deviation of latency data to the average latency within the unit time window; the latency feature set refers to the set formed by integrating the average latency and the latency jitter coefficient; sliding window statistics refers to the method of collecting bandwidth data and performing statistics by sliding the window in chronological order with a fixed duration; bandwidth utilization rate refers to the ratio of real-time data transmission bandwidth to the maximum supported bandwidth of the node; peak bandwidth ratio refers to the ratio of the peak bandwidth within a unit time to the maximum supported bandwidth of the node; bandwidth fluctuation frequency refers to the number of times bandwidth data exceeds the preset stable range within a unit time; effective transmission refers to the ratio of effective data transmission volume to total transmission volume within a unit time; the bandwidth performance index set refers to the set formed by integrating bandwidth utilization rate, peak bandwidth ratio, bandwidth fluctuation frequency, and effective transmission rate; normalization processing refers to the processing method of converting computing power data of different ranges to a unified numerical range; CPU utilization rate is... The table lists the following metrics for a node's performance: CPU utilization rate (RRR) = CPU resources currently used by a node to total CPU resources; memory utilization rate (RRR) = memory resources currently used by a node to total memory resources; I / O operations per second (IOPS) = the number of I / O operations a node can complete per unit of time; task queue length = the number of tasks currently waiting to be processed by a node; computing power load assessment value = the integrated and normalized real-time node computing power data combined with CPU utilization, memory utilization rate, IOPS, and task queue length; three-dimensional collaborative analysis model = an analysis model used to integrate latency, bandwidth, and computing power data, achieving collaborative analysis of multi-dimensional data through spatiotemporal correlation mapping; comprehensive load index = a comprehensive value reflecting the overall load level of a node calculated by the three-dimensional collaborative analysis model; load status level = a node load category based on the comprehensive load index; and storage resource real-time performance data package = a data set containing complete node performance and load information, formed by the structured integration of latency feature set, bandwidth performance index set, computing power load assessment value, comprehensive load index, and load status level.

[0061] In this application example, firstly, the real-time response latency data acquired in S101 is processed through step 1021: First, a fixed unit time window is set, and all real-time response latency data of a certain storage node within the window are summarized. The average latency is calculated according to the formula "average latency = sum of latency data within the unit time window ÷ number of latency data acquisitions within the unit time window", where the sum of latency data within the unit time window refers to the cumulative value of all real-time response latencies within the window, and the number of latency data acquisitions within the unit time window refers to the total number of times latency data is acquired within the window. Then, the latency jitter coefficient is calculated according to the formula "latency jitter coefficient = standard deviation of latency data within the unit time window ÷ average latency within the unit time window", where the standard deviation of latency data within the unit time window refers to the degree to which the latency data within the window deviates from the average latency. The two are integrated to form the latency feature set of the node, providing basic data for the latency dimension for subsequent multi-dimensional collaborative analysis.

[0062] Next, based on the processing logic of latency data in step 1021, the real-time data transmission bandwidth data obtained in S101 is processed synchronously through step 1022: First, the sliding window duration and the maximum supported bandwidth of the node are set. Bandwidth data is statistically analyzed segment by segment in the sliding window. The bandwidth utilization rate is calculated according to the formula "bandwidth utilization = real-time data transmission bandwidth within the sliding window ÷ maximum supported bandwidth of the node". The peak bandwidth ratio is calculated according to the formula "peak bandwidth ratio = peak bandwidth within the sliding window ÷ maximum supported bandwidth of the node", where the peak bandwidth within the sliding window refers to the maximum real-time bandwidth within the window. The bandwidth fluctuation frequency is statistically analyzed according to the formula "bandwidth fluctuation frequency = number of times the bandwidth exceeds the preset stable range within the sliding window". The effective transmission rate is calculated according to the formula "effective data transmission amount within the sliding window ÷ total data transmission amount within the sliding window". These four indicators are integrated to form the bandwidth performance indicator set of the node, which together with the latency feature set constitutes the two core dimensions of multi-dimensional analysis data.

[0063] Next, the real-time node computing power data obtained by S101 is processed through steps 1023: First, a minimum-maximum normalization method is used, and the real-time node computing power data is converted to a unified range of 0-1 according to the formula "normalized computing power data = (original real-time node computing power value - minimum real-time computing power value of all nodes) ÷ (maximum real-time computing power value of all nodes - minimum real-time computing power value of all nodes)". Here, the original real-time node computing power value refers to the computing power data of a certain node obtained by S101, and the minimum / maximum real-time computing power value of all nodes refers to the minimum / maximum value among the original computing power values ​​of all nodes participating in the current task stage; then, the CPU utilization rate of the node is obtained, that is, the ratio of occupied CPU resources to total CPU resources, and the memory utilization rate. This involves calculating the ratio of occupied memory resources to total memory resources, the number of input / output operations per second, and the task queue length, converting all four data types to a range of 0-1. Based on preset weights—normalized computing power data 0.3, CPU utilization 0.2, memory utilization 0.2, IOPS 0.15, and task queue length 0.15, with a weight sum of 1—the computing power load assessment value is calculated using the formula: "Computing power load assessment value = (normalized computing power data × 0.3) + (CPU utilization × 0.2) + (memory utilization × 0.2) + (IOPS × 0.15) + (task queue length × 0.15)". This completes the processing of the computing power dimension data, forming the third core dimension of multi-dimensional analysis.

[0064] Subsequently, a multi-dimensional collaborative analysis was conducted through 1024 steps: A three-dimensional collaborative analysis model was invoked. First, in the time dimension, the unit time window of step 1021, the sliding window of step 1022, and the data collection time of computing power data in step 1023 were aligned to ensure that the three types of data reflected the node status within the same time period. Then, in the spatial dimension, the latency feature set, bandwidth performance index set, and computing power load assessment value of the same node were correlated, according to the model's preset dimension weights: latency feature set 0.3, bandwidth performance index set 0.3, and computing power load assessment value 0.4, and calculated according to the formula... The comprehensive load index is calculated as follows: "Comprehensive load index = (latency feature set score × 0.3) + (bandwidth performance index set score × 0.3) + (computing power load assessment value × 0.4)". The latency feature set score and bandwidth performance index set score are 0-1 scores converted by the model based on the specific values ​​of the feature set and index set. Then, according to the preset comprehensive load index range, i.e. 0-0.4 is low load, 0.4-0.7 is medium load, and 0.7-1 is high load, the load status level of the node is determined, realizing the quantification and classification of the overall load of the node.

[0065] Finally, all data is integrated through 1025 steps: the latency feature set, bandwidth performance index set, computing power load assessment value, comprehensive load index, and load status level of each node are integrated according to the preset structured format of "node identifier - latency feature set - bandwidth performance index set - computing power load assessment value - comprehensive load index - load status level" to form a real-time performance data packet of storage resources containing complete performance and load information of each node. This data packet will serve as the core input for the performance analysis of the "artificial intelligence model corresponding to the task stage" in the subsequent S103 step, providing data support for obtaining the target value of storage resource allocation.

[0066] In practical applications, during the data import phase of distributed AI model training, the following data is collected based on S101 for the storage nodes A, B, and C participating in this phase: real-time response latency data: 25 milliseconds for node A, 30 milliseconds for node B, and 22 milliseconds for node C; real-time data transmission bandwidth data: 69.3 MB / s for node A, 62.1 MB / s for node B, and 75.5 MB / s for node C; and real-time node computing power data: 28.8 for node A, 24 for node B, and 32 for node C. Collaborative processing is then carried out.

[0067] First, when processing real-time response latency data, a 1-minute time window is set. Latency data is collected 12 times within each node window. The sum of latency data for node A within the time window is 25 × 12 = 300 milliseconds, collected 12 times. For node B, the sum is 30 × 12 = 360 milliseconds, collected 12 times. For node C, the sum is 22 × 12 = 264 milliseconds, collected 12 times. Using the formula "Average latency = Sum of latency data within a time window ÷ Number of latency data collections within a time window", the average latency for node A is calculated to be 300 ÷ 12 = 25 milliseconds, for node B it is 360 ÷ 12 = 30 milliseconds, and for node C it is 264 ÷ 12 = 22 milliseconds. Next, the delay jitter coefficient of each node is calculated according to the formula "delay jitter coefficient = standard deviation of delay data within a unit time window ÷ average delay within a unit time window". Where the standard deviation of delay data of node A is 2 milliseconds, the jitter coefficient of node A is 2 ÷ 25 = 0.08; the standard deviation of delay data of node B is 3 milliseconds, the jitter coefficient of node B is 3 ÷ 30 = 0.1; and the standard deviation of delay data of node C is 1.5 milliseconds, the jitter coefficient of node C is 1.5 ÷ 22 ≈ 0.07. The delay feature sets of the three nodes are formed respectively for subsequent collaborative analysis.

[0068] Next, when processing real-time data transmission bandwidth data, a 10-second sliding window and a maximum supported bandwidth of 100MB / s for each node were set. The total data transmission volume within the sliding window for each node was calculated using the formula: "Total data transmission volume = Real-time data transmission bandwidth × Sliding window duration". Specifically, the total data transmission volume for node A was 69.3MB / s × 10s = 693MB, for node B it was 62.1MB / s × 10s = 621MB, and for node C it was 75.5MB / s × 10s = 755MB. Then, the amount of invalid data for each node was determined: because node A has moderate transmission link stability, the amount of invalid data was 693MB × 2%. The transmission link stability of node B is slightly worse, with invalid data amounting to 621MB × 3% = 18.63MB. The transmission link stability of node C is the best, with invalid data amounting to 755MB × 1% = 7.55MB. Then, using the formula "Effective data transmission amount = Total data transmission amount - Invalid data amount", the effective data transmission amount of each node is derived as follows: Effective data transmission amount of node A = 693MB - 13.86MB = 679.14MB, Effective data transmission amount of node B = 621MB - 18.63MB = 602.37MB, and Effective data transmission amount of node C = 755MB - 7.55MB = 747.45MB.

[0069] Then, the bandwidth utilization of each node is calculated according to the formula "bandwidth utilization = real-time data transmission bandwidth within the sliding window ÷ maximum supported bandwidth of the node". The real-time data transmission bandwidth within the sliding window of node A is 69.3 MB / s, so the bandwidth utilization of node A is 69.3 ÷ 100 = 0.693, or 69.3%. The real-time data transmission bandwidth within the sliding window of node B is 62.1 MB / s, so the bandwidth utilization of node B is 62.1 ÷ 100 = 0.621, or 62.1%. The real-time data transmission bandwidth within the sliding window of node C is 75.5 MB / s, so the bandwidth utilization of node C is 75.5 ÷ 100 = 0.755, or 75.5%. The peak bandwidth percentage of each node is calculated using the formula "Peak bandwidth percentage = Peak bandwidth within the sliding window ÷ Maximum supported bandwidth of the node". Node A's peak bandwidth within the sliding window is 72 MB / s, so its peak bandwidth percentage is 72 ÷ 100 = 0.72, or 72%. Node B's peak bandwidth within the sliding window is 65 MB / s, so its peak bandwidth percentage is 65 ÷ 100 = 0.65, or 65%. Node C's peak bandwidth within the sliding window is 78 MB / s, so its peak bandwidth percentage is 78 ÷ 100 = 0.78, or 78%. The bandwidth fluctuation frequency of each node is calculated using the formula "Bandwidth fluctuation frequency = Number of times the bandwidth exceeds the preset stable range within the sliding window". The preset stable range is 65-75 MB / s. The statistics show that node A exceeded the limit 3 times, so its bandwidth fluctuation frequency is 3. Node B exceeded the limit 2 times, so its bandwidth fluctuation frequency is 2. Node C exceeded the limit 1 time, so its bandwidth fluctuation frequency is 1. The effective transmission rate of each node is calculated using the formula "Effective transmission rate = Effective data transmission volume within the sliding window ÷ Total data transmission volume within the sliding window". The effective transmission rate of node A is 679.14MB ÷ 693MB × 100% = 98%, the effective transmission rate of node B is 602.37MB ÷ 621MB × 100% = 97%, and the effective transmission rate of node C is 747.45MB ÷ 755MB × 100% = 99%. These are integrated to form a set of bandwidth performance indicators for subsequent collaborative analysis.

[0070] Next, when processing the real-time node computing power data, the normalized computing power data of each node is calculated according to the formula: "Normalized computing power data = (Original real-time node computing power value - Minimum real-time computing power value of all nodes) ÷ (Maximum real-time computing power value of all nodes - Minimum real-time computing power value of all nodes)". The minimum real-time computing power value of all nodes is 24, and the maximum is 32. The original real-time node computing power value of node A is 28.8, so the normalized computing power data of node A is (28.8-24) ÷ (32-24) = 0.6. The original real-time node computing power value of node B is 24, so the normalized computing power data of node B is (24-24) ÷ (32-24) = 0. The original real-time node computing power of C is 32. The normalized computing power of node C is (32-24) ÷ (32-24) = 1. Next, the CPU utilization of each node is obtained: node A's CPU utilization is 0.5, node B's is 0.55, and node C's is 0.45. The memory usage of each node is also obtained: node A's is 0.5, node B's is 0.6, and node C's is 0.4. Finally, the number of input / output operations per second (IOPS) for each node is obtained: node A's IOPS is 0.7, node B's is 0.6, and node C's is 0.4. The input / output operands are 0.8. The task queue lengths of each node are obtained: node A's task queue length is 0.4, node B's is 0.5, and node C's is 0.3. The computational load assessment value for each node is calculated using the formula: "Computational load assessment value = (Normalized computational load data × 0.3) + (CPU utilization × 0.2) + (Memory utilization × 0.2) + (IOPS × 0.15) + (Task queue length × 0.15)". The computational load assessment value for node A is (0.6 × 0.3) + (0.5 × 0.2) + (0.5 × 0.2) + (0.7 × 0.15) + (0.4 × 0.15) + (0.4 × 0.15) + (0.6 × 0.3) + (0.5 × 0.2) + (0.7 × 0.15) + (0.4 × 0.15) + (0.4 × 0.15) + (0.5 × 0.2) + (0.5 × 0.2) + (0.7 ...5 × 0.2) + (0.7 × 0.15) + (0.5 × 0.15) + (0.5 × 0.2) + (0.5 × 0.2) + (0.5 × 0.2) + ( 5) = 0.18 + 0.1 + 0.1 + 0.105 + 0.06 = 0.545, the computing load assessment value of node B is (0 × 0.3) + (0.55 × 0.2) + (0.6 × 0.2) + (0.6 × 0.15) + (0.5 × 0.15) = 0 + 0.11 + 0.12 + 0.09 + 0.075 = 0.395, the computing load assessment value of node C is (1 × 0.3) + (0.45 × 0.2) + (0.4 × 0.2) + (0.8 × 0.15) + (0.3 × 0.15) = 0.3 + 0.09 + 0.08 + 0.12 + 0.045 = 0.635, which are used for subsequent collaborative analysis.

[0071] Subsequently, a three-dimensional collaborative analysis model was invoked. Its time dimension aligns with the data from a 1-minute unit window and a 10-second sliding window, while its spatial dimension correlates the latency feature set, bandwidth performance index set, and computing load assessment value of each node. The comprehensive load index of each node was calculated using the formula: "Comprehensive Load Index = (Latency Feature Set Score × 0.3) + (Bandwidth Performance Index Set Score × 0.3) + (Computing Load Assessment Value × 0.4)". The latency feature set scores for node A were 0.6, for node B 0.65, and for node C 0.5; the bandwidth performance index set scores for node A were 0.65, for node B 0.6, and for node C 0.72. Therefore, the comprehensive load index for node A was (0.6 × 0.3) + (0 × 0.4) + (0 × 0.5) + (0 × 0.6 ... 0.65×0.3)+(0.545×0.4)=0.18+0.195+0.218=0.593; The comprehensive load index of node B is (0.65×0.3)+(0.6×0.3)+(0.395×0.4)=0.195+0.18+0.158=0.533; The comprehensive load index of node C is (0.5×0.3)+(0.72×0.3)+(0.635×0.4)=0.15+0.216+0.254=0.62; Based on the comprehensive load index range, namely 0-0.4 low load, 0.4-0.7 medium load, and 0.7-1 high load, nodes A, B, and C are all determined to be medium load level.

[0072] Finally, following the format of "node identifier - latency feature set - bandwidth performance index set - computing power load assessment value - comprehensive load index - load status level", all data of nodes A, B, and C are integrated to form a real-time performance data packet for storage resources. This data packet will be passed to the subsequent S103 step for performance analysis with the artificial intelligence model corresponding to the data import stage to obtain the target value for storage resource allocation for each node.

[0073] The aforementioned S102 overall solution, through targeted processing of three types of dispersed basic data, transforms single-dimensional latency, bandwidth, and computing power data into multi-dimensional data containing feature sets, indicator sets, and evaluation values, avoiding the one-sidedness of single data. Then, a three-dimensional collaborative analysis model is used to achieve spatiotemporal correlation of multi-dimensional data, generating a comprehensive load index and load status level that reflects the overall state of the nodes, making the node load situation more intuitive and comprehensive. Finally, all data is structured and integrated to form a real-time performance data package for storage resources. This not only eliminates the differences in magnitude and format between different data sets but also provides unified, complete, and accurate input information for the performance analysis of the artificial intelligence model in the subsequent S103 step, effectively ensuring the accuracy and efficiency of the subsequent analysis process. Simultaneously, it makes the performance and load status of storage nodes quantifiable and assessable, providing a scientific decision-making basis for the subsequent dynamic allocation of storage resources.

[0074] S103. Based on the artificial intelligence model corresponding to the task stage, perform performance analysis on the real-time performance data packet of the storage resources to obtain the target value for storage resource allocation.

[0075] Optionally, step S103 involves performing performance analysis on the real-time performance data packets of the storage resources based on the artificial intelligence model corresponding to the task stage, to obtain the target value for storage resource allocation, including:

[0076] Step 1031: Obtain the requirement characteristics corresponding to the current task stage. The requirement characteristics include the task data processing scale, read and write operation frequency, data persistence requirements, and service quality level.

[0077] Step 1032: Associate and integrate the real-time performance data packet of the storage resources with the demand characteristics to obtain the data to be analyzed.

[0078] Step 1033: Using the artificial intelligence model corresponding to the current task stage, perform performance analysis on the data to be analyzed to obtain the performance target instructions for each storage node to meet the requirements of the current task stage.

[0079] Step 1033 may specifically include the following steps: Using the comparison module in the artificial intelligence model corresponding to the current task stage, the real-time performance data packets of storage resources in the data to be analyzed are compared with the demand characteristics to obtain the difference information between each storage node and the demand characteristics in terms of response speed, data transmission capability, and processing capability; Using the analysis module in the artificial intelligence model, the difference information is correlated with the real-time load status of the corresponding storage node to determine the shortage situation causing the difference; Using the adjustment module in the artificial intelligence model, the adjustment direction of each storage node in terms of response speed, data transmission capability, and processing capability is determined for the shortage situation; Using the integration module in the artificial intelligence model, all the adjustment directions are integrated to form the performance target instructions for each storage node to meet the requirements of the current task stage.

[0080] Step 1034: Based on the response speed adaptation requirements in the performance target instructions, combine the real-time response latency data with the real-time load status to obtain a response latency threshold. Based on the data transmission capability adaptation requirements in the performance target instructions, combine the real-time data transmission bandwidth data with the real-time load status to obtain a data transmission bandwidth quota. Based on the processing capability adaptation requirements in the performance target instructions, combine the real-time node computing power data with the real-time load status to obtain a computing power support quota.

[0081] Step 1035: Summarize all the aforementioned response latency thresholds, data transmission bandwidth quotas, and computing power support quotas to obtain the target value for storage resource allocation for each storage node.

[0082] In the above scheme, the current task stage refers to the current processing stage in the execution of the target task, and the demand for storage resources varies at different stages; demand characteristics refer to the set of specific requirements parameters for storage resources at the current task stage, including task data processing scale, read and write operation frequency, data persistence requirements, and service quality level; data to be analyzed refers to the data formed by associating and integrating real-time performance data packets of storage resources with demand characteristics, which is used to provide unified analysis input for the artificial intelligence model; the artificial intelligence model corresponding to the current task stage refers to a dedicated analysis model trained for the demand characteristics of the current stage, which includes four functional modules: comparison module, analysis module, adjustment module, and integration module; the comparison module is the sub-module in the model used to compare the performance of storage resources with the demand characteristics; the difference information refers to the gap data between each storage node and the demand characteristics in terms of response speed, data transmission capacity, and processing capacity; the analysis module is the sub-module in the model used to associate the difference information with the node load status; the shortage situation refers to the situation that causes the difference between node performance and demand. The model defines the specific types of resource shortages; the adjustment module is a sub-module in the model used to determine the direction of node performance optimization; the adjustment direction refers to the performance dimension that a node needs to improve or optimize in response to shortage situations; the integration module is a sub-module in the model used to summarize the adjustment directions; the performance target instruction refers to the structured instruction containing the adjustment directions of each node; the response latency threshold refers to the maximum response latency standard that a node needs to achieve, determined based on the response speed requirements in the performance target instruction and the real-time load status of the node; the data transmission bandwidth quota refers to the minimum bandwidth quota that a node can be allocated, determined based on the transmission capacity requirements in the performance target instruction and the real-time load status of the node; the computing power support quota refers to the minimum computing power standard that a node needs to achieve, determined based on the processing capacity requirements in the performance target instruction and the real-time computing power data and load status of the node; and the storage resource allocation target value refers to the resource allocation benchmark value that each node needs to meet, formed by summarizing the response latency threshold, data transmission bandwidth quota, and computing power support quota of all nodes, and is used to formulate subsequent scheduling strategies.

[0083] In this application example, firstly, the requirement characteristics of the current task stage are obtained through step 1031: First, based on the stage definition of the target task, such as the data import stage and the computation processing stage of distributed AI model training, the core resource requirement direction of this stage is clarified. For example, the data import stage requires high bandwidth and low latency. Then, the specific requirement parameters of this stage are extracted through the task management system, including the task data processing scale, i.e., the total amount of data to be processed in the current stage, the read and write operation frequency, i.e. the number of read and write requests to the storage node per unit time, the data persistence requirements, i.e. the attribute of data to be stored long-term or temporarily, and the service quality level, i.e. the minimum standard for response latency and transmission bandwidth. These parameters are integrated to form the requirement characteristics, which will serve as the benchmark for subsequent judgment on whether the node performance is suitable for the task requirements.

[0084] Next, based on the demand characteristics obtained in step 1031, the data to be analyzed is generated through step 1032: First, the key performance indicators of each node in the real-time performance data packet of storage resources output in step S102 are extracted, such as response latency, data transmission bandwidth, and computing load assessment value. Then, these performance indicators are bound one by one with the corresponding parameters in the demand characteristics. For example, the node response latency is bound to the latency requirement of the service quality level, and the node bandwidth is bound to the bandwidth requirement of the service quality level, forming the data to be analyzed that contains the correspondence between "actual node performance and task requirement standard". This data will serve as the analysis input of the artificial intelligence model corresponding to the current task stage, ensuring that the model can conduct analysis by combining the node status and task requirements at the same time.

[0085] Then, using the data to be analyzed generated in step 1032 as input, the performance target instructions are generated through the artificial intelligence model in step 1033: First, the model's comparison module is invoked to compare the performance indicators of each node in the data to be analyzed with the corresponding standards of the required characteristics item by item, outputting the difference information. If the performance meets the standard, it is marked "no difference"; if it does not meet the standard, the degree of difference is quantified. Second, the model's analysis module is invoked to correlate the difference information with the real-time load status of the corresponding node, i.e., the load status level output in step S102, such as low load, medium load, and high load. If a difference exists, the short circuit causing the difference is identified. In the first step, if there are any shortages, such as excessive load leading to excessive response latency or insufficient bandwidth resources leading to insufficient transmission capacity, then it is determined that there is "no shortage". The second step is to call the model's adjustment module to determine the performance adjustment direction of each node for the shortage situation. For example, if the shortage situation is "insufficient bandwidth", the adjustment direction is "increase bandwidth quota"; if there is no shortage, the adjustment direction is "maintain current performance and optimize stability". The third step is to call the model's integration module to summarize the adjustment directions of each node in the format of "node identifier-adjustment direction" to form a performance target instruction. This instruction will be used to guide the calculation of subsequent resource performance target parameters.

[0086] Next, based on the performance target instructions generated in step 1033, the resource performance target parameters for each node are calculated in step 1034: First, addressing the response speed adaptation requirement in the performance target instructions, the response latency threshold is calculated using the formula: Response Latency Threshold = Real-time Response Latency Data × Load Influence Coefficient, combining the real-time response latency data obtained in step S101 and the real-time load status in step S102. The load influence coefficient is set according to the node's load status; for example, a higher coefficient corresponds to a higher load, reserving more performance redundancy to cope with load fluctuations. Second, addressing the data transmission capability adaptation requirement in the performance target instructions, the data transmission bandwidth quota is calculated using the formula: Data Transmission Bandwidth Quota = Real-time Data Transmission Bandwidth Data × Load Influence Coefficient, combining the real-time node computing power data in step S101 and the real-time load status in step S102. Third, addressing the processing capability adaptation requirement in the performance target instructions, the computing power support quota is calculated using the formula: Computing Power Support Quota = Real-time Node Computing Power Data × Load Influence Coefficient, combining the real-time node computing power data in step S101 and the real-time load status in step S102. These three parameters together constitute the resource performance target that the node must meet.

[0087] Finally, the target value for storage resource allocation is generated through step 1035: First, according to the "node identifier", the response latency threshold, data transmission bandwidth quota, and computing power support quota of each node are collected; then, compared with the demand characteristics obtained in step 1031, it is checked whether each parameter meets the minimum requirements of the service quality level, such as whether the response latency threshold is ≤ the upper limit of latency in the demand, and whether the bandwidth quota is ≥ the lower limit of bandwidth in the demand. If not, it is adjusted to meet the requirements; finally, the three types of parameters of each node are integrated into structured data to form the target value for storage resource allocation of each node. This target value will serve as the core basis for formulating resource scheduling strategies through dynamic allocation optimization algorithms in the subsequent step S104.

[0088] In practical applications, during the data import phase of distributed AI model training, step S103 is performed on storage nodes A, B, and C participating in this phase. First, the required characteristics for this phase are determined: the task data processing scale is 100GB, the read / write operation frequency is 500 data write requests per second, the data persistence requirement is temporary storage (data does not need to be retained long-term), and the service quality level is a response latency ≤30 milliseconds and a data transmission bandwidth ≥60MB / s. These characteristics will serve as the benchmark for judging the node performance suitability.

[0089] Next, the real-time performance data packet of storage resources output in step S102 is correlated. This data packet contains the following information: Node A: average latency 25 milliseconds, data transmission bandwidth 69.3 MB / s, and computing load assessment value 0.545; Node B: average latency 30 milliseconds, data transmission bandwidth 62.1 MB / s, and computing load assessment value 0.395; Node C: average latency 22 milliseconds, data transmission bandwidth 75.5 MB / s, and computing load assessment value 0.635. The performance indicators of each node are then bound to the corresponding demand characteristics to form the data to be analyzed. For example, the data to be analyzed for Node B includes the correspondence between "average latency 30 milliseconds - demand response latency ≤ 30 milliseconds, data transmission bandwidth 62.1 MB / s - demand bandwidth ≥ 60 MB / s". This data to be analyzed will be used for the subsequent performance analysis of the artificial intelligence model.

[0090] Then, the corresponding artificial intelligence model in the data import stage is invoked. The first step involves analyzing the data through a comparison module. Node A has an average latency of 25-30 milliseconds and a bandwidth of 69.3 MB / s-60 MB / s; Node B has an average latency of 30-30 milliseconds and a bandwidth of 62.1 MB / s-60 MB / s; Node C has an average latency of 22-30 milliseconds and a bandwidth of 75.5 MB / s-60 MB / s. All nodes meet the performance standards, and the output difference information is "The response speed and data transmission capabilities of each node meet the required characteristics." The second step involves analyzing the data through the analysis module. The load status of each node determined in step S102 is classified as "no resource shortage". The third step uses the adjustment module to determine the adjustment direction of each node as "maintain the current performance level and optimize data transmission stability". The fourth step uses the integration module to summarize the adjustment directions and form the performance target instructions: "Node A: Maintain the current response latency and bandwidth, optimize transmission stability; Node B: Maintain the current response latency and bandwidth, optimize transmission stability; Node C: Maintain the current response latency and bandwidth, optimize transmission stability". This instruction will be used to calculate the resource performance target parameters of each node.

[0091] Subsequently, based on the performance target instructions and combined with the real-time data obtained in step S101: Node A real-time response latency 25 milliseconds, bandwidth 69.3 MB / s, computing power 28.8; Node B real-time response latency 30 milliseconds, bandwidth 62.1 MB / s, computing power 24; Node C real-time response latency 22 milliseconds, bandwidth 75.5 MB / s, computing power 32, and combined with the medium load status in step S102, the load impact coefficient corresponding to the medium load is set to 1.05. The load impact coefficient is set according to the load level. High load requires a higher coefficient to reserve redundancy. Setting the medium load to 1.05 can balance performance and redundancy requirements. The response latency threshold for each node is calculated using the formula "Response Latency Threshold = Real-time Response Latency Data × Load Impact Coefficient". The response latency threshold for node A is 25 × 1.05 = 26.25 milliseconds, and for node B it is 30 × 1.05 = 31.5 milliseconds. Since 31.5 milliseconds is greater than the minimum requirement of 30 milliseconds, the response latency threshold for node B is adjusted to 30 milliseconds. The response latency threshold for node C is 22 × 1.05 = 23.1 milliseconds. The data transmission bandwidth quota for each node is calculated using the formula "Data Transmission Bandwidth Quota = Real-time Data Transmission Bandwidth Data × Load Impact Coefficient". The response latency threshold for node A is... The data transmission bandwidth quota is approximately 69.3 × 1.05 = 72.77 MB / s for node B, 62.1 × 1.05 = 65.21 MB / s for node C, and 75.5 × 1.05 = 79.28 MB / s for node C. The computing power support quota for each node is calculated using the formula "Computing power support quota = Real-time node computing power data × Load impact coefficient". The calculated computing power support quotas are: 28.8 × 1.05 = 30.24 for node A, 24 × 1.05 = 25.2 for node B, and 32 × 1.05 = 33.6 for node C.

[0092] Finally, the response latency threshold, data transmission bandwidth quota, and computing power support limit of each node are summarized to form the target values ​​for storage resource allocation for each node. These are: Node A: 26.25 milliseconds, 72.77 MB / s, 30.24 MB / s; Node B: 30 milliseconds, 65.21 MB / s, 25.2 MB / s; Node C: 23.1 milliseconds, 79.28 MB / s, 33.6 MB / s. These target values ​​for storage resource allocation will be passed to the subsequent S104 step, used to formulate the resource scheduling strategy for each storage node through a dynamic allocation optimization algorithm.

[0093] The aforementioned S103 overall solution first clarifies the demand characteristics of the current task phase, providing a clear target orientation for resource analysis and avoiding blind analysis divorced from task requirements. Then, it integrates the demand characteristics with node performance data into the data to be analyzed, ensuring that the input of the AI ​​model includes both the actual state of the nodes and the relevance to task requirements, thus improving the analysis's focus. Through a dedicated AI model with multi-module functions, it accurately identifies the discrepancies between node performance and requirements, determines the causes of these discrepancies, and identifies adjustment directions, avoiding the subjectivity and bias of manual analysis. Based on the adjustment direction and node load status, it calculates response latency thresholds, bandwidth quotas, and computing power limits, ensuring that resource targets meet both the minimum requirements and allow for redundancy due to load fluctuations, thus ensuring the feasibility of the targets. Finally, the summarized storage resource allocation target values ​​provide a quantitative and precise benchmark for subsequent resource scheduling strategies, effectively ensuring that subsequent resource allocation aligns with the current task phase requirements and the actual state of the nodes, avoiding resource mismatch or performance failures, and laying a scientific target foundation for the dynamic allocation of storage resources.

[0094] S104. Using a dynamic allocation optimization algorithm, a resource scheduling strategy is formulated based on the target value of the storage resource allocation, and a real-time control signal corresponding to the resource scheduling strategy is generated. Based on the real-time control signal, the dynamic allocation and scheduling of storage resources among each storage node is executed.

[0095] Optionally, step S104 involves formulating a resource scheduling strategy based on the storage resource allocation target value using a dynamic allocation optimization algorithm, and generating a real-time control signal corresponding to the resource scheduling strategy, including:

[0096] Step 1041: Based on the storage resource allocation target value, and combined with the hardware specifications and task load characteristics of each storage node, calculate the resource requirement list of each storage node under the current task stage. The resource requirement list includes: storage capacity, read / write throughput, storage resource type and quantity.

[0097] Step 1042: Collect real-time available resource data of all storage nodes, summarize them to form a global available resource list, compare the resource demand list of each storage node with the global available resource list item by item, and quantify the supply and demand difference of various resources based on the comparison results. The part of demand exceeding supply is the resource gap, and the part of supply exceeding demand is the resource surplus. The global available resource list represents the total amount of actual available resources of all storage nodes.

[0098] Step 1043: Based on the supply and demand difference of the various types of resources, determine the resource allocation order of each storage node through a dynamic allocation optimization algorithm.

[0099] Step 1043 specifically includes the following steps: Based on the supply and demand difference of various resources, determine the corresponding allocation gap or surplus information, wherein the allocation gap or surplus information includes: resource gap or surplus type, scale, and scope of impact; classify the allocation gap or surplus information according to the type of response latency resources, data transmission bandwidth resources, and computing power resources to obtain the gap degree or surplus degree corresponding to each type of resource; compare the gap degree or surplus degree of each type of resource with the corresponding preset resource importance level to determine the allocation priority of the corresponding resource in the current task stage; extract the demand value of each type of resource in each storage node and the urgency level identifier of the current task carried by each storage node from the resource demand list; associate the demand value with the urgency level identifier, and based on the association result, sort each storage node under the same resource according to the preset sorting rules to obtain the node allocation sub-order corresponding to the resource; and integrate all the node allocation sub-orders in series according to the allocation priority to obtain the resource allocation order of each storage node.

[0100] Step 1044: Adjust the resource requirement list of each storage node according to the resource allocation order to form a resource scheduling strategy for each storage node. The resource scheduling strategy includes: resource allocation path, adjustment range and implementation time.

[0101] Step 1044 specifically includes the following steps: Following the resource allocation order, select the resource type with the highest current priority to obtain the resource to be processed; according to the node allocation sub-order corresponding to the resource to be processed, select the current processing node sequentially; obtain the current processing node's demand value for the resource to be processed, the total supply of the resource to be processed, and the cumulative amount already allocated to other nodes; calculate the difference between the total supply and the cumulative amount to obtain the remaining allocable amount; if the demand value is less than or equal to the remaining allocable amount, then use the demand value as the actual allocation amount for the current processing node; or, if the demand value is greater than the remaining allocable amount, then, combined with the real-time load status of the current processing node and the upper limit of the carrying capacity of the resource to be processed, use a value that does not exceed the remaining allocable amount and does not exceed the upper limit of the carrying capacity as the actual allocation amount for the current processing node; record the actual allocation amount as the final allocation result of the current processing node for the resource to be processed; integrate the final allocation results of each storage node in various types of resources to form a resource scheduling strategy for each storage node.

[0102] Step 1045: Perform rule conversion on the resource scheduling strategy to obtain instruction parameters, encode the instruction parameters according to a preset signal format, and generate a real-time control signal corresponding to the resource scheduling strategy.

[0103] In the above scheme, hardware specifications refer to the hardware configuration parameters of each storage node; task load characteristics refer to the load characteristics of the node under the current task stage; resource requirement list refers to the detailed node resource requirements formed by integrating storage capacity, read / write throughput, storage resource type and quantity; storage capacity refers to the total amount of data that the node needs to carry; read / write throughput refers to the amount of read / write data that the node needs to complete per unit time; real-time available resource data refers to the resource data that is not currently occupied by each node; global available resource list refers to the system-level resource list formed by summarizing the real-time available resource data of all nodes; supply-demand gap refers to the difference between resource supply and demand obtained by comparing the resource requirement list and the global available resource list, with the part of demand exceeding supply being the resource gap, and the part of supply exceeding demand being the resource surplus; dynamic allocation optimization algorithm refers to the algorithm used to determine the resource allocation order and optimize the resource allocation efficiency; preset resource importance level refers to the level of resource importance based on the task. The resource priority is preset for each stage of requirements; urgency level identifier refers to the label indicating the urgency level of the current task undertaken by each node; node allocation sub-order refers to the order in which nodes are allocated according to sorting rules under the same resource type; resource allocation order refers to the system-level resource allocation process formed by connecting all node allocation sub-orders according to resource priority; resource scheduling strategy refers to the node resource allocation scheme that includes resource allocation path, adjustment range, and implementation time; resource allocation path refers to the transmission path of resources from surplus nodes to shortage nodes; adjustment range refers to the specific value of resource allocation or transfer; implementation time refers to the execution time point of resource allocation; instruction parameters refer to the machine-recognizable parameters converted from the resource scheduling strategy; preset signal format refers to the system-agreed control signal encoding format; real-time control signal refers to the execution signal generated after encoding instruction parameters according to the preset signal format, used to trigger each node to perform resource allocation operations.

[0104] In practical applications, during the data import phase of distributed AI model training, step S104 is performed on storage nodes A, B, and C. This process needs to be correlated with the output results of S101, S102, and S103. Specifically, S103 has determined the target values ​​for storage resource allocation for each node: Node A has a bandwidth of 72.77 MB / s and a computing power of 30.24; Node B has a bandwidth of 65.21 MB / s and a computing power of 25.2; and Node C has a bandwidth of 79.28 MB / s and a computing power of 33.6. S102 has determined that each node is in a medium-load state. S101 has collected basic real-time data for each node, including: Node A has a real-time response latency of 25 milliseconds, a bandwidth of 69.3 MB / s, and a computing power of 28.8; Node B has a real-time response latency of 30 milliseconds, a bandwidth of 62.1 MB / s, and a computing power of 24; and Node C has a real-time response latency of 22 milliseconds, a bandwidth of 75.5 MB / s, and a computing power of 32.

[0105] First, calculate the resource requirements: Assume all nodes have identical hardware specifications, with a maximum supported bandwidth of 100MB / s, a maximum storage capacity of 500GB, and a CPU with 32 cores. The current task data processing scale is 100GB. The data volume is allocated according to the node's capacity: node A handles 40%, node B handles 35%, and node C handles 25%. Calculate the storage capacity of each node using the formula: "Storage capacity = Task data processing scale × Node capacity ratio". The results are: Node A = 100GB × 40% = 40GB, Node B = 100GB × 35% = 35GB, and Node C = 100GB. B × 25% = 25GB; According to the rule that read / write throughput is consistent with bandwidth quota, the read / write throughput of node A is 72.77MB / s, the read / write throughput of node B is 65.21MB / s, and the read / write throughput of node C is 79.28MB / s; The storage resource quantity corresponds to the target value determined in S103, that is, node A has a bandwidth of 72.77MB / s and a computing power requirement of 30.24, node B has a bandwidth of 65.21MB / s and a computing power requirement of 25.2, and node C has a bandwidth of 79.28MB / s and a computing power requirement of 33.6; Integrate the above data to form a resource requirement list for each node, which will be used for supply and demand comparison later.

[0106] Next, the supply-demand gap is calculated: real-time available resource data for each node is collected, resulting in: Node A: available bandwidth 25MB / s, available computing power 8, available storage 100GB; Node B: available bandwidth 30MB / s, available computing power 10, available storage 80GB; Node C: available bandwidth 20MB / s, available computing power 12, available storage 120GB. The available bandwidth, available computing power, and available storage of each node are then aggregated to form a global available resource list, resulting in: global available bandwidth = 25 + 30 + 20 = 75MB / s, global available computing power = 8 + 10 + 12 = 30, and global... Available storage = 100 + 80 + 120 = 300 GB; Calculate the supply and demand difference: Global bandwidth requirement = 72.77 + 65.21 + 79.28 = 217.26 MB / s, bandwidth gap = 217.26 - 75 = 142.26 MB / s; Global computing power requirement = 30.24 + 25.2 + 33.6 = 89.04, computing power gap = 89.04 - 30 = 59.04; Global storage requirement = 40 + 35 + 25 = 100 GB; Storage surplus = 300 - 100 = 200 GB; This supply and demand difference will be used to determine the resource allocation order.

[0107] Next, the resource allocation order is determined: During the data import phase, the preset resource importance level is set as "bandwidth resources > computing power resources > storage resources." Based on the supply-demand difference, allocation gap or surplus information is generated, resulting in severe bandwidth shortage, severe computing power shortage, and slight storage surplus. Subsequently, the resource demand values ​​for each node are extracted, showing that node C's bandwidth demand is 79.28 > node A's is 72.77 > node B's is 65.21, and node C's computing power demand is 33.6 > node A's is 30.24 > node B's is 2. 5.2 Node A's storage requirement is 40 > Node B's storage requirement is 35 > Node C's storage requirement is 25. The task urgency level of each node is medium-urgent. According to the sorting rule of "largest requirement value first", the node allocation sub-order is obtained as follows: bandwidth is C→A→B, computing power is C→A→B, and storage is A→B→C. The resource allocation order is formed by concatenating the sub-orders according to resource priority: first allocate bandwidth according to C→A→B, then allocate computing power according to C→A→B, and finally allocate storage according to A→B→C. This allocation order will be used to formulate resource scheduling strategies.

[0108] Next, a resource scheduling strategy is formed: bandwidth resources are processed according to the resource allocation order, with node C being the first to be processed. The remaining allocable amount for each node is calculated using the formula "Remaining Allocable Amount = Total Supply - Cumulative Allocated Amount". The total supply is the global available bandwidth of 75MB / s, and the initial allocated cumulative amount is 0. Therefore, the remaining allocable amount for node C is 75 - 0 = 75MB / s. Since node C's bandwidth requirement of 79.28MB / s is greater than 75MB / s, and considering the bandwidth capacity limit of 80MB / s corresponding to the load, the actual allocated amount for node C is 75MB / s, and the cumulative allocated amount is updated to 75MB / s. Because the bandwidth requirements of node A (72.77 MB / s) and node B (65.21 MB / s) both exceed the remaining allocable resources, it is planned to allocate 72.77 MB / s and 65.21 MB / s respectively from the reserved resource pool. Next, computing power resources will be processed, with node C being the first to be processed. The total supply is 30 globally available computing power, with 0 already allocated, leaving 30 - 0 = 30 available resources. Since node C's computing power requirement of 33.6 is greater than 30, and considering the computing power capacity limit of 32 corresponding to its load, the actual allocated resource for node C is 12, and the already allocated cumulative resource is updated to 12. Similarly, allocation will be done for node A. Available computing power is 8, and the cumulative allocated power is updated to 20. Node B is allocated 10 available computing power, and the cumulative allocated power is updated to 30. The computing power gap for node A is calculated as 30.24 - 8 = 22.24, for node B as 25.2 - 10 = 15.2, and for node C as 33.6 - 12 = 21.6, which will be supplemented from the reserved resource pool. Storage resources are then processed and allocated according to demand: 40GB for node A, 35GB for node B, and 25GB for node C, all within the globally available storage range. The above data is then integrated to form the resource scheduling for each node. The strategy is as follows: Node A: "Allocation path: Reserved resource pool → A; Adjustment range: Bandwidth +72.77MB / s, Computing power +22.24, Storage +40GB; Implementation time: T+10 seconds"; Node B: "Allocation path: Reserved resource pool → B; Adjustment range: Bandwidth +65.21MB / s, Computing power +15.2, Storage +35GB; Implementation time: T+10 seconds"; Node C: "Allocation path: Reserved resource pool → C; Adjustment range: Bandwidth +4.28MB / s, Computing power +21.6, Storage +25GB; Implementation time: T+10 seconds". This scheduling strategy will be subsequently converted into real-time control signals.

[0109] Finally, real-time control signals are generated: the resource scheduling strategies of each node are converted into instruction parameters, resulting in the following instruction parameters for node A: "Allocation path: Pool→A; Bandwidth adjustment value: 72.77; Computing power adjustment value: 22.24; Storage adjustment value: 40; Implementation time: T+10"; for node B: "Allocation path: Pool→B; Bandwidth adjustment value: 65.21; Computing power adjustment value: 15.2; Storage adjustment value: 35; Implementation time: T+10"; and for node C: "Allocation path: Pool→C; Bandwidth adjustment value: 4.28; Computing power adjustment value: 21.6; Storage adjustment value: 25; Implementation time: T+10". The instruction parameters for each node are then encoded according to a preset JSON format, resulting in the following encoding for node A: "{"node":"A","path":"Pool→A","bandwidth":72.77,"compu The code snippet shows the process: `{"node":"B","path":"Pool→B","bandwidth":65.21,"computing":15.2,"storage":35,"time":"T+10"}`. Node B is encoded as `{"node":"B","path":"Pool→C","bandwidth":4.28,"computing":21.6,"storage":25,"time":"T+10"}`. A real-time control signal corresponding to the scheduling strategy is generated and sent to each storage node and the reserved resource pool. This signal will subsequently trigger each node to dynamically allocate and schedule storage resources, completing the connection with subsequent execution stages.

[0110] The aforementioned S104 overall solution calculates a resource requirement list by combining the target value of storage resource allocation with node hardware specifications and task load characteristics, clarifying the specific types and scale of resource requirements for each node and avoiding ambiguity in resource allocation direction. By collecting real-time available resources and summarizing a global list, and then comparing it with the requirement list to statistically analyze the supply and demand gap, it clearly understands the current status of system resource supply and demand, providing objective data support for resource allocation decisions. Based on a dynamic allocation optimization algorithm, it determines the allocation order by combining the degree of resource gap and task stage priority, ensuring that resources prioritize meeting key needs and improving the targeting of resource allocation. It adjusts the requirement list according to the allocation order and clarifies the allocation path, adjustment range, and implementation time, giving the resource scheduling strategy executable details and avoiding vague strategies. Through rule conversion and encoding, it generates real-time control signals to ensure that scheduling instructions can be accurately identified and triggered by the system, realizing a closed loop from target value to actual resource allocation.

[0111] The following is a complete example for steps S101 to S104, such as Figure 2 As shown, the first step involves sending 10 training data read requests of 200MB each to each of the three nodes, recording the time taken for each request, and calculating the real-time response latency: Node A's 10 requests took 22 milliseconds, 23 milliseconds, 22 milliseconds, 24 milliseconds, 23 milliseconds, 22 milliseconds, 23 milliseconds, 24 milliseconds, 22 milliseconds, and 23 milliseconds respectively. Using the formula "Average latency = Sum of latency data within a unit time window ÷ Number of latency data collections within a unit time window", the average latency of each node is calculated. The average latency of node A is (22 + 23 + 22 + 24 + 23 + 22 + 23 + 24 + 22 + 23) ÷ 10 = 22.8 milliseconds. Similarly, the average latency of node B is calculated to be 30.2 milliseconds, and that of node C is 21.5 milliseconds. The second step involves recording the data transmitted by each node within one minute using a traffic monitoring tool, and calculating the real-time data transmission bandwidth: Node A received 3.8GB of data and sent 0.2GB of status data. The data volume is converted to MB (1GB = 1024M). (B) The total transmission volume is calculated as 3.8 × 1024 + 0.2 × 1024 = 4096 MB. Using the formula "Real-time data transmission bandwidth = Total transmission volume ÷ Monitoring time", the bandwidth of each node is calculated as follows: Node A bandwidth = 4096 MB ÷ 60 seconds ≈ 68.27 MB / s. Similarly, the bandwidth of node B is calculated to be 61.8 MB / s, and the bandwidth of node C is 74.9 MB / s. The third step involves obtaining the hardware parameters of each node and calculating the real-time node computing power data, including the total number of CPU cores of node A. With 32 cores, a CPU idle rate of 60%, and 18GB of available memory, the real-time node computing power of each node is calculated using the formula "Real-time node computing power = Total CPU cores × CPU idle rate + Available memory × Memory computing power coefficient". The memory computing power coefficient is preset to 0.8. The real-time node computing power of node A is 32 × 60% + 18 × 0.8 = 19.2 + 14.4 = 33.6. Similarly, the real-time node computing power of node B is calculated to be 24.5, and the real-time node computing power of node C is 35.2.

[0112] Next, the first step, based on the acquired three types of real-time data, is to process the real-time response latency data to generate a latency feature set: Latency data for each node is statistically analyzed using a 1-minute unit window. The standard deviation of latency data for node A is 1.2 milliseconds. The latency jitter coefficient for each node is calculated using the formula "latency jitter coefficient = standard deviation of latency data within a unit time window ÷ average latency within a unit time window". The latency jitter coefficient for node A is 1.2 ÷ 22.8 ≈ 0.053. Similarly, the latency jitter coefficients for node B (0.08) and node C (0.04) are calculated. The average latency and jitter coefficients are then integrated to form the latency feature set for each node. The second step is to process the real-time data transmission bandwidth data to generate a bandwidth performance index set: Latency data for each node is statistically analyzed using a 10-second sliding window. The bandwidth data of the nodes shows that node A has a maximum supported bandwidth of 100MB / s. Using the formula "bandwidth utilization = real-time data transmission bandwidth within the sliding window ÷ maximum supported bandwidth of the node," the bandwidth utilization of each node is calculated. Node A's bandwidth utilization is 68.27 ÷ 100 = 0.6827, or 68.27%. Since node A's peak bandwidth within the sliding window is 71MB / s, the peak bandwidth percentage of each node is calculated using the formula "peak bandwidth percentage = peak bandwidth within the sliding window ÷ maximum supported bandwidth of the node." Node A's peak bandwidth percentage is 71%. The number of times node A exceeds the stable range of "65-75MB / s" is counted, i.e., the bandwidth fluctuation frequency is 2 times. The proportion of effective data transmission to total transmission volume is then calculated. That is, the effective transmission rate is 98%. Similarly, calculate the bandwidth utilization, peak bandwidth ratio, fluctuation frequency, and effective transmission rate of nodes B and C, and integrate them to form a set of bandwidth performance indicators for each node. The third step is to process the real-time node computing power data to generate computing power load assessment values: First, normalize the computing power of all nodes according to the formula "normalized computing power data = (original real-time node computing power value - minimum real-time computing power value of all nodes) ÷ (maximum real-time computing power value of all nodes - minimum real-time computing power value of all nodes)". The minimum computing power value of all nodes is 24.5 and the maximum value is 35.2. We get the normalized computing power of node A = (33.6-24.5) ÷ (35.2-24.5) ≈ 0.85. Then, obtain the CPU utilization rate of each node to get the computing power load assessment value of node A. The CPU utilization rate is 40% for node A, 45% for node B, and 35% for node C. The memory utilization rate of each node is obtained: 45% for node A, 50% for node B, and 40% for node C. The IOPS are obtained: 0.75 for node A, 0.65 for node B, and 0.8 for node C. The task queue length is obtained: 0.35 for node A, 0.4 for node B, and 0.3 for node C. According to the formula "Computational load assessment value = Normalized computing power data × 0.3 + CPU utilization rate × 0.2 + Memory utilization rate × 0.3", the calculated value is calculated as follows:The computational load assessment value of each node is calculated using the formula: 2 + IOPS × 0.15 + Task queue length × 0.15. The computational load assessment value for node A is calculated as: 0.85 × 0.3 + 0.4 × 0.2 + 0.45 × 0.2 + 0.75 × 0.15 + 0.35 × 0.15 = 0.255 + 0.08 + 0.09 + 0.1125 + 0.0525 = 0.59. Similarly, the assessment values ​​for node B (0.62) and node C (0.55) are calculated. The fourth step involves using the three-dimensional collaborative analysis model to generate a comprehensive load index and load status level. The comprehensive load index is calculated according to the formula: "Comprehensive Load Index = Latency Feature Set Score × 0.3 + Bandwidth Performance Index Set Score × 0.3 + Computational Load Assessment Value × 0.4". The overall load index is calculated as follows: Node A has a latency feature set score of 0.55 and a bandwidth performance index score of 0.62. The overall load index for node A is calculated as: 0.55 × 0.3 + 0.62 × 0.3 + 0.59 × 0.4 = 0.165 + 0.186 + 0.236 = 0.587. Based on the overall load index range (0-0.4 for low load, 0.4-0.7 for medium load, and 0.7-1 for high load), node A is determined to be under medium load. Similarly, nodes B and C are also determined to be under medium load. The fifth step integrates the latency feature set, bandwidth performance index set, computing power load assessment value, overall load index, and load status level to form a real-time storage resource performance data package. This data package will be used for the AI ​​model performance analysis in the third step.

[0113] The first step is to determine the requirements of the current task phase: task data processing scale of 100GB, read / write operation frequency of 500 times / second, data persistence requirement of temporary storage, service quality level response latency ≤30 milliseconds and data transmission bandwidth ≥60MB / s; the second step is to associate the generated real-time performance data package of storage resources with the requirements to form the data to be analyzed: binding the average latency of each node with the latency requirement of the service quality level, and binding the bandwidth utilization rate with the bandwidth requirement, ensuring that the data to be analyzed contains the correspondence between "node performance - requirement standard"; the third step is to call the artificial intelligence model corresponding to the data import phase to process the data to be analyzed: the model comparison module compares the node performance with the requirement standard item by item, and judges... Node A has a latency of 22.8 milliseconds ≤ 30 milliseconds and a bandwidth of 68.27 MB / s ≥ 60 MB / s, which is considered indistinguishable. Similarly, nodes B and C are also deemed indistinguishable, and the difference information "Performance of each node meets requirements" is output. Subsequently, the model analysis module correlates the load status of each node to determine the shortage situation, resulting in "No resource shortage" for each node. The model adjustment module, for the absence of shortage, determines the adjustment direction for each node as "Maintain current performance and optimize transmission stability". The model integration module summarizes the adjustment directions, forming the performance target instructions as follows: "Node A: Maintain latency and bandwidth, optimize transmission stability; Node B: Maintain latency and bandwidth, optimize transmission stability; Node C: Maintain latency and bandwidth, optimize transmission stability". The first step is to calculate resource performance target parameters based on performance target instructions: The load impact coefficient corresponding to the medium load is set to 1.05. The response latency threshold for each node is calculated using the formula "Response Latency Threshold = Real-time Response Latency Data × Load Impact Coefficient". The response latency thresholds for node A are: 22.8 × 1.05 = 23.94 milliseconds; and for node B: 30.2 × 1.05 = 31.71 milliseconds. Since 31.71 milliseconds exceeds the 30 millisecond minimum requirement, the response latency threshold for node B is adjusted to 30 milliseconds. The response latency threshold for node C is: 21.5 × 1.05 = 22.575 milliseconds. The response latency threshold for node C is calculated using the formula "Data Transmission Bandwidth Quota = Real-time Data Transmission Bandwidth Data × Load Impact Coefficient". The data transmission bandwidth quota for each node is calculated using the following formula: Node A: 68.27 × 1.05 ≈ 71.68 MB / s; Node B: 61.8 × 1.05 ≈ 64.89 MB / s; Node C: 74.9 × 1.05 ≈ 78.65 MB / s. The computing power support quota for each node is then calculated using the formula: "Computing power support quota = Real-time node computing power data × Load impact coefficient". The results are: Node A: 33.6 × 1.05 = 35.28; Node B: 24.5 × 1.05 ≈ 25.73; Node C: 35.2 × 1.05 ≈ 37.0; The fifth step is to summarize the response latency threshold, data transmission bandwidth quota, and computing power support limit of each node to form the target value for storage resource allocation.

[0114] Finally, based on the storage resource allocation target value generated in step three, the first step calculates the resource requirement list for each node: according to the task load ratio, i.e., node A carries 40%, B carries 35%, and C carries 25%, the storage capacity of each node is calculated using the formula "storage capacity = task data processing scale × load ratio". The results are: Node A's storage capacity = 100GB × 40% = 40GB, Node B's storage capacity = 100GB × 35% = 35GB, and Node C's storage capacity = 100GB × 25% = 25GB. Combining this with bandwidth quota and computing power quota, a resource requirement list is formed: Node A "storage capacity 40GB, read / write throughput 71.68MB / s". s. Resource type and quantity: Bandwidth 71.68MB / s, computing power 35.28, storage space 40GB; similarly, resource requirement lists for nodes B and C are formed; the second step is to collect real-time available resource data for all nodes and summarize them to form a global available resource list: Node A available bandwidth 25MB / s, available computing power 8, available storage 100GB; Node B available bandwidth 30MB / s, available computing power 10, available storage 80GB; Node C available bandwidth 20MB / s, available computing power 12, available storage 120GB; global available bandwidth = 25 + 30 + 20 = 75MB / s; global available computing power = 8 + 10 + 12 = 32; global available storage = 100 + 80 + 120 = 300GB; The third step is to compare the resource demand list with the global available resource list and calculate the supply-demand gap: Calculate the total global resource supply for each node using the formula "Supply-demand gap = Total resource demand - Total global resource supply". Global bandwidth demand = 71.68 + 64.89 + 78.65 = 215.22 MB / s, bandwidth gap = 215.22 - 75 = 140.22 MB / s; Global computing power demand = 35.28 + 25.73 + 37.0 = 98.01, computing power gap = 98.01 - 32 = 66.01; Global storage demand = 40 + 35 + 25 = 100GB, storage surplus = 30 0-100=200GB; Step 4: Determine the resource allocation order and actual allocation amount based on the supply and demand difference: Preset the resource importance level "bandwidth resources > computing power resources > storage resources", and according to the sorting rule of "larger demand value first", obtain the bandwidth allocation sub-order C→A→B, computing power allocation sub-order C→A→B, and storage allocation sub-order A→B→C, which are connected in series to form the resource allocation order, which is "first allocate bandwidth according to C→A→B, then allocate computing power according to C→A→B, and finally allocate storage according to A→B→C"; Step 5: Calculate the actual allocation amount of each node according to the formula "remaining allocable amount = total supply amount - cumulative allocated amount": When allocating bandwidth, the demand of node C is 78.With a total supply of 75MB / s and an initial allocation of 0, the remaining allocable capacity is calculated as 75 - 0 = 75MB / s. Since node C's demand exceeds the remaining allocable capacity, and its load bandwidth capacity is capped at 80MB / s, node C's actual allocation is 75MB / s, and the cumulative allocation is updated to 75. Nodes A and B's bandwidth demands exceed the remaining allocable capacity, so plans are in place to allocate 71.68MB / s and 64.89MB / s respectively from the reserved resource pool. Similarly, computing power is allocated, resulting in node C receiving 12, A receiving 8, and B receiving 10. Shortfalls are then filled from the reserved pool and stored accordingly. Allocating resources according to demand, there are no shortfalls for any node. The sixth step integrates and forms a resource scheduling strategy and generates a real-time control signal: Node A scheduling strategy: "Allocation path: Reserved resource pool → A; Adjustment magnitude: Bandwidth + 71." 68MB / s, computing power +35.28, storage +40GB; implementation time: T+10 seconds; similarly, scheduling strategies for nodes B and C are formed, and the scheduling strategies are converted into instruction parameters, such as node A's "allocation path: Pool→A; bandwidth adjustment value: 71.68; computing power adjustment value: 35.28; storage adjustment value: 40; implementation time: T+10", encoded in JSON format to obtain real-time control signals, such as {"node":"A","path":"Pool→A","bandwidth":71.68,"computing":35.28,"storage":40,"time":"T+10"}, which are then sent to each node and the reserved resource pool to execute dynamic allocation of storage resources among the storage nodes.

[0115] Figure 3 This is a schematic diagram illustrating a specific implementation of an artificial intelligence-based dynamic storage resource allocation system provided in this application. (Refer to...) Figure 3 The system may include:

[0116] The acquisition module 31 is used to acquire the real-time response latency data, real-time data transmission bandwidth data and real-time node computing power data of each storage node at the current task stage of the target task.

[0117] Processing module 32 is used to collaboratively process the real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node.

[0118] Analysis module 33 is used to perform performance analysis on the real-time performance data packets of the storage resources based on the artificial intelligence model corresponding to the task stage, so as to obtain the target value of storage resource allocation.

[0119] The generation module 34 is used to formulate a resource scheduling strategy based on the storage resource allocation target value through a dynamic allocation optimization algorithm, generate a real-time control signal corresponding to the resource scheduling strategy, and execute the dynamic allocation and scheduling of storage resources among storage nodes based on the real-time control signal.

[0120] The AI-based dynamic allocation system for storage resources in this application is used to implement the aforementioned AI-based dynamic allocation method for storage resources. Therefore, the specific implementation of the AI-based dynamic allocation system for storage resources can be found in the embodiment section of the AI-based dynamic allocation method for storage resources above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0121] like Figure 4 As shown, this application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described artificial intelligence-based dynamic allocation methods for storage resources.

[0122] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described artificial intelligence-based dynamic allocation methods for storage resources. In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0123] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the AI-based dynamic allocation method for storage resources.

[0124] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0125] The foregoing has provided a detailed description of the artificial intelligence-based dynamic allocation method and system for storage resources provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for dynamically allocating storage resources based on artificial intelligence, characterized in that, include: Obtain real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task; The real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data are processed together to obtain a real-time performance data packet of storage resources that includes the real-time load status of each storage node. Based on the artificial intelligence model corresponding to the task stage, the real-time performance data packet of the storage resources is analyzed to obtain the target value for storage resource allocation. The dynamic allocation optimization algorithm is used to formulate a resource scheduling strategy based on the target value of the storage resource allocation, and generate a real-time control signal corresponding to the resource scheduling strategy. Based on the real-time control signal, the dynamic allocation and scheduling of storage resources among the storage nodes are executed. The step of formulating a resource scheduling strategy based on the target value of storage resource allocation using a dynamic allocation optimization algorithm, and generating a real-time control signal corresponding to the resource scheduling strategy, includes: Based on the storage resource allocation target value, and combined with the hardware specifications and task load characteristics of each storage node, calculate the resource requirement list of each storage node under the current task stage. The resource requirement list includes: storage capacity, read / write throughput, storage resource type and quantity. Collect real-time available resource data from all storage nodes and compile them into a global available resource list. Compare the resource demand list of each storage node with the global available resource list item by item. Based on the comparison results, quantify and calculate the supply and demand difference of various resources. The part where demand exceeds supply is the resource gap, and the part where supply exceeds demand is the resource surplus. Based on the supply and demand difference of the various resources, the resource allocation order of each storage node is determined through a dynamic allocation optimization algorithm. According to the resource allocation order, the resource requirement list of each storage node is adjusted to form a resource scheduling strategy for each storage node. The resource scheduling strategy includes: resource allocation path, adjustment range and implementation time. The resource scheduling strategy is transformed into rules to obtain instruction parameters. The instruction parameters are then encoded according to a preset signal format to generate a real-time control signal corresponding to the resource scheduling strategy.

2. The method according to claim 1, characterized in that, The process of determining the resource allocation order of each storage node based on the supply-demand gap of the various resources using a dynamic allocation optimization algorithm includes: Based on the supply and demand gaps of various resources, the corresponding allocation gaps or surplus information is determined. The allocation gaps or surplus information includes: the type, scale, and scope of impact of the resource gaps or surpluses. The allocation gap or surplus information is classified according to the type of response latency resources, data transmission bandwidth resources, and computing power resources to obtain the gap or surplus degree corresponding to each type of resource. The shortage or surplus of each type of resource is compared with the corresponding preset resource importance level to determine the allocation priority of the corresponding resource in the current task stage. Extract the required value of each type of resource in each storage node and the urgency level of the current task carried by each storage node from the resource requirement list; The required value is associated with the urgency level identifier. Based on the association result, the storage nodes under the same resource are sorted according to the preset sorting rules to obtain the node allocation sub-order corresponding to the resource. According to the allocation priority, all the node allocation sub-orders are concatenated and integrated to obtain the resource allocation order of each storage node.

3. The method according to claim 1, characterized in that, The step of adjusting the resource requirement list of each storage node according to the resource allocation order to form a resource scheduling strategy for each storage node includes: According to the resource allocation order, the resource type with the highest priority is selected to obtain the resource to be processed. According to the sub-allocation order of the nodes corresponding to the resource to be processed, the current processing node is selected in turn. Obtain the current processing node's demand for the resource to be processed, the total supply of the resource to be processed, and the cumulative amount already allocated to other nodes. Calculate the difference between the total supply and the cumulative amount to obtain the remaining allocable amount. If the demand value is less than or equal to the remaining allocable amount, then the demand value is taken as the actual allocation amount of the current processing node; or, if the demand value is greater than the remaining allocable amount, then, in combination with the upper limit of the capacity of the resource to be processed in the real-time load status of the current processing node, the value that does not exceed the remaining allocable amount and does not exceed the upper limit of the capacity is taken as the actual allocation amount of the current processing node. The actual allocation amount is recorded as the final allocation result of the current processing node to the resource to be processed. The final allocation results of each storage node in various types of resources are integrated to form a resource scheduling strategy for each storage node.

4. The method according to claim 1, characterized in that, The process of performing performance analysis on the real-time performance data packets of the storage resources based on the artificial intelligence model corresponding to the task stage to obtain the target value for storage resource allocation includes: Obtain the requirement characteristics corresponding to the current task stage, including the task data processing scale, read and write operation frequency, data persistence requirements, and service quality level; The real-time performance data packets of the storage resources are associated and integrated with the demand characteristics to obtain the data to be analyzed; By using the artificial intelligence model corresponding to the current task stage, the performance of the data to be analyzed is performed to obtain the performance target instructions for each storage node to meet the requirements of the current task stage. Based on the response speed adaptation requirements in the performance target instructions, the real-time response latency data is combined with the real-time load status to obtain the response latency threshold. Based on the data transmission capability adaptation requirements in the performance target instructions, the real-time data transmission bandwidth data is combined with the real-time load status to obtain the data transmission bandwidth quota. Based on the processing capability adaptation requirements in the performance target instructions, the real-time node computing power data is combined with the real-time load status to obtain the computing power support quota. The target value for storage resource allocation for each storage node is obtained by summarizing all the aforementioned response latency thresholds, data transmission bandwidth quotas, and computing power support quotas.

5. The method according to claim 4, characterized in that, The step involves using the artificial intelligence model corresponding to the current task stage to perform performance analysis on the data to be analyzed, and obtaining performance target instructions for each storage node to meet the requirements of the current task stage, including: By using the comparison module in the artificial intelligence model corresponding to the current task stage, the real-time performance data packets of storage resources in the data to be analyzed are compared with the demand characteristics to obtain the difference information between each storage node and the demand characteristics in terms of response speed, data transmission capability, and processing capability. The analysis module in the artificial intelligence model correlates the difference information with the real-time load status of the corresponding storage node to determine the shortage situation that caused the difference. The adjustment module in the artificial intelligence model determines the adjustment direction of each storage node in terms of response speed, data transmission capacity, and processing capacity in response to the shortage situation. The integration module in the artificial intelligence model integrates all the aforementioned adjustment directions to form performance target instructions for each storage node to meet the needs of the current task stage.

6. The method according to claim 1, characterized in that, The step of collaboratively processing the real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node includes: Based on the real-time response delay data, the average delay and delay jitter coefficient within a unit time window are calculated to form a delay feature set; The real-time data transmission bandwidth data is statistically analyzed using a sliding window to calculate bandwidth utilization, peak bandwidth ratio, bandwidth fluctuation frequency, and effective transmission rate, thereby generating a set of bandwidth performance indicators. The real-time node computing power data is normalized, and a computing power load assessment value is constructed by combining the node CPU utilization, memory usage, number of input / output operations per second, and task queue length. Using a three-dimensional collaborative analysis model, the latency feature set, the bandwidth performance index set, and the computing power load evaluation value are spatiotemporally correlated and mapped to generate a comprehensive load index for each storage node, and the load status level is determined based on the comprehensive load index. The latency feature set, bandwidth performance index set, computing power load assessment value, comprehensive load index and load status level of each storage node are structurally integrated to form a real-time performance data packet for storage resources.

7. A dynamic storage resource allocation system based on artificial intelligence, characterized in that, A method for implementing a dynamic allocation of storage resources based on artificial intelligence as described in claim 1, comprising: The acquisition module is used to acquire real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task. The processing module is used to collaboratively process the real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node. The analysis module is used to perform performance analysis on the real-time performance data packets of the storage resources based on the artificial intelligence model corresponding to the task stage, so as to obtain the target value for storage resource allocation. The generation module is used to formulate a resource scheduling strategy based on the storage resource allocation target value through a dynamic allocation optimization algorithm, generate a real-time control signal corresponding to the resource scheduling strategy, and execute the dynamic allocation and scheduling of storage resources among storage nodes based on the real-time control signal.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the AI-based dynamic allocation method for storage resources as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the artificial intelligence-based dynamic allocation method for storage resources as described in any one of claims 1 to 6.

Images