Dynamic storage resource allocation method and system based on artificial intelligence

Abstract

The invention provides a storage resource dynamic allocation method and system based on artificial intelligence, and relates to the technical field of storage resource dynamic allocation, and the method comprises the steps: obtaining the real-time response delay, data transmission bandwidth and node computing power data of each storage node in the current stage of a target task, and carrying out the cooperative processing of the data, thereby achieving the dynamic allocation of storage resources. Obtaining a storage resource real-time performance data packet containing the real-time load state of each storage node; analyzing the data packet based on an artificial intelligence model corresponding to the task stage to obtain a storage resource allocation target value; through a dynamic allocation optimization algorithm, a resource scheduling strategy is formulated according to a target value, a real-time control signal is generated, and dynamic allocation and scheduling of storage resources among storage nodes are executed accordingly, so that the dynamic allocation and scheduling of the storage resources among the storage nodes can be realized based on an artificial intelligence-based storage resource dynamic allocation method, and the dynamic allocation and scheduling of the storage resources can be realized through data co-processing, AI analysis adaptive to a task stage and the dynamic optimization algorithm. Accurate and dynamic allocation and scheduling of storage resources meeting target task requirements are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dynamic allocation of storage resources, and in particular to a storage resource dynamic allocation method and system based on artificial intelligence. BACKGROUND

[0002] In cloud computing, big data processing, artificial intelligence training and other scenarios that rely on the cooperation of multiple storage nodes, a target task usually includes different task stages such as data collection, calculation processing and result output. The response speed, data transmission capacity and calculation support capacity of the storage nodes required by each stage are significantly different. For example, the storage nodes need to have high data transmission bandwidth in the data collection stage, and high computing power support in the calculation processing stage. If the storage resource allocation cannot be dynamically adjusted according to the task stage, it is easy to cause task execution to be stuck or resources to be idle, so there is an urgent need for a storage resource dynamic allocation scheme that can meet the requirements of the task stage.

[0003] At present, in the existing storage resource allocation technology, resource allocation templates are usually preset, such as allocating resources according to the number of storage nodes or based on a single real-time index, such as the current bandwidth occupancy rate of the node or the CPU utilization rate to trigger resource adjustment. The specific operation is to first collect the basic performance data of each storage node regularly, and then compare it with the preset load threshold to determine whether adjustment is needed. In the adjustment process, fixed algorithms such as polling and weighted average are usually used to determine the allocation ratio of resources among nodes to maintain the load of each node in a roughly balanced state.

[0004] However, the existing technology has a core disadvantage, which is mainly that the resource allocation strategy is not formulated in combination with the specific stage characteristics of the target task, but only adjusted according to a general template or a single index, resulting in a disconnection between resource allocation and the actual needs of the task stage. For example, when the task is in the data transmission stage, it needs high bandwidth support, but the existing technology may allocate computing power resources according to the average proportion; when the task enters the calculation stage, it needs high computing power, but the existing technology still maintains high bandwidth allocation, causing resource mismatch and failing to achieve precise adaptation of resources to task needs, thereby affecting the efficiency of task execution or causing resource waste. SUMMARY

[0005] The purpose of the present application is to provide a storage resource dynamic allocation method and system based on artificial intelligence to solve the problem of inaccurate adaptation of storage resources to task needs in the prior art.

[0006] To solve the above technical problems, in a first aspect, the present application provides a storage resource dynamic allocation method based on artificial intelligence, comprising: obtaining real-time response delay data, real-time data transmission bandwidth data and real-time node computing power data of each storage node in the task stage of a target task; The real-time response delay data, the real-time data transmission bandwidth data and the real-time node computing power data are cooperatively processed to obtain a storage resource real-time performance data packet containing real-time load states of each storage node; Based on an artificial intelligence model corresponding to a task phase, performance analysis is performed on the storage resource real-time performance data packet to obtain a storage resource allocation target value;

[0007] Through a dynamic allocation optimization algorithm, a resource scheduling strategy is formulated based on the storage resource allocation target value, and a real-time control signal corresponding to the resource scheduling strategy is generated. Based on the real-time control signal, dynamic allocation and scheduling of storage resources among the storage nodes are performed.

[0008] Optionally, the dynamic allocation optimization algorithm is used to formulate a resource scheduling strategy based on the storage resource allocation target value, and a real-time control signal corresponding to the resource scheduling strategy is generated, including: According to the storage resource allocation target value, the hardware specifications and task load characteristics of each storage node are combined to calculate a resource requirement list of each storage node under the current task phase. The resource requirement list includes storage capacity, read-write throughput, storage resource type and quantity. Real-time available resource data of all storage nodes is collected to form a global available resource list. The resource requirement list of each storage node is compared with the global available resource list item by item, and the supply-demand difference of each type of resource is quantitatively calculated according to the comparison result. The part that exceeds the supply is the resource gap, and the part that exceeds the demand is the resource surplus. Based on the supply-demand difference of each type of resource, a dynamic allocation optimization algorithm is used to determine the resource allocation sequence of each storage node. According to the resource allocation sequence, the resource requirement list of each storage node is adjusted to form a resource scheduling strategy for each storage node. The resource scheduling strategy includes resource allocation path, adjustment amplitude and implementation time. The resource scheduling strategy is converted into a rule to obtain an instruction parameter. The instruction parameter is encoded according to a preset signal format to generate a real-time control signal corresponding to the resource scheduling strategy.

[0009] Optionally, the dynamic allocation optimization algorithm is used to formulate a resource scheduling strategy based on the storage resource allocation target value, and a real-time control signal corresponding to the resource scheduling strategy is generated, including: Based on the supply-demand difference of each type of resource, the corresponding allocation gap or surplus information is determined. The allocation gap or surplus information includes resource gap or surplus type, size and influence range. According to the types of response delay resources, data transmission bandwidth resources, and computing power resources, the allocation gap or surplus information is classified and processed to obtain the gap degree or surplus degree of each type of resource; The gap degree or surplus degree of each type of resource is compared with the corresponding preset resource importance level to determine the allocation priority of the corresponding resource in the current task stage. The required values of each type of resource in each storage node and the urgency identifiers of the current carrying tasks of each storage node are extracted from the resource requirement list. The required values are associated with the urgency identifiers, and based on the association result, each storage node under the same resource is sorted according to a preset sorting rule to obtain the corresponding node allocation sub-sequence of the corresponding resource. According to the allocation priority, all the node allocation sub-sequences are concatenated and integrated to obtain the resource allocation sequence of each storage node.

[0010] In a second aspect, the present application provides a storage resource dynamic allocation system based on artificial intelligence, comprising: An acquisition module is configured to acquire real-time response delay data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node in a task stage of a target task. A processing module is configured to cooperatively process the real-time response delay data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a storage resource real-time performance data packet containing real-time load states of each storage node. An analysis module is configured to perform performance analysis on the storage resource real-time performance data packet based on an artificial intelligence model corresponding to the task stage to obtain a storage resource allocation target value. A generation module is configured to develop a resource scheduling strategy based on the storage resource allocation target value through a dynamic allocation optimization algorithm, generate a real-time control signal corresponding to the resource scheduling strategy, and perform dynamic allocation and scheduling of storage resources among each storage node based on the real-time control signal.

[0011] In a third aspect, the present application provides an electronic device, comprising: A memory is configured to store a computer program. A processor is configured to execute the computer program to implement the steps of the storage resource dynamic allocation method based on artificial intelligence of the first aspect described above.

[0012] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the storage resource dynamic allocation method based on artificial intelligence of the first aspect described above.

[0013] The storage resource dynamic allocation method based on artificial intelligence provided by the present application obtains real-time response delay data, real-time data transmission bandwidth data and real-time node computing power data of each storage node in the task stage of the target task; the real-time response delay data, the real-time data transmission bandwidth data and the real-time node computing power data are processed cooperatively to obtain a storage resource real-time performance data packet containing the real-time load state of each storage node; based on an artificial intelligence model corresponding to the task stage, the storage resource real-time performance data packet is analyzed to obtain a storage resource allocation target value; through a dynamic allocation optimization algorithm, a resource scheduling strategy is formulated based on the storage resource allocation target value, and a real-time control signal corresponding to the resource scheduling strategy is generated, and based on the real-time control signal, the dynamic allocation and scheduling of storage resources among each storage node are performed.

[0014] The technical scheme of the present application has the following beneficial effects: The present application ensures that the collected data is strongly associated with the current stage of the target task and the actual running state of each storage node, avoids irrelevant data interference, provides accurate and targeted basic data support for subsequent storage resource allocation, and guarantees the input effectiveness of the subsequent processing link; through the cooperative integration of multiple data, the performance and real-time load of each storage node are comprehensively reflected, forming complete and coherent performance data results, providing comprehensive and reliable input information for subsequent artificial intelligence model analysis; relying on the artificial intelligence model adapted to the task stage, the performance analysis process closely matches the specific needs of the current task stage, avoiding the problem of insufficient adaptability of general analysis, ensuring that the output storage resource allocation target value accurately matches the task stage characteristics and node actual performance, and providing a scientific and demand-oriented basis for subsequent resource scheduling; by converting the abstract allocation target value into a specific executable resource scheduling strategy, and then executing it through the real-time control signal, the dynamic adaptation of resources among each storage node is realized, ensuring that the resource allocation is synchronized with the task demand and the node state in real time, effectively improving the storage resource utilization efficiency and the adaptability of task execution.

[0015] Further, the application calculates the resource demand list of each node in the current task stage according to the storage resource allocation target value, in combination with the hardware specifications and task load characteristics of each storage node; then collects the real-time available resource data of all storage nodes and aggregates to form a global available resource list, compares the resource demand list of each node with the global available resource list item by item, and quantitatively calculates the supply-demand difference of each type of resource according to the comparison result; then, based on the supply-demand difference of each type of resource, the resource allocation sequence of each storage node is determined through a dynamic allocation optimization algorithm; then, according to the allocation sequence, the resource demand list of each node is adjusted to form a resource scheduling strategy for each storage node, including resource allocation path, adjustment range and implementation time; finally, the resource scheduling strategy is converted to obtain instruction parameters, and the instruction parameters are encoded according to the preset signal format to generate real-time control signals corresponding to the resource scheduling strategy.

[0016] The application ensures the quantification and clarity of the resource demand of each storage node by calculating the explicit resource demand list, avoids allocation deviation caused by ambiguous demand; accurately grasps the supply and demand situation of each type of resource in the system by collecting and aggregating the global available resources and comparing the supply and demand difference, and provides objective data support for resource allocation; ensures the order and rationality of the resource allocation process by determining the resource allocation sequence through a dynamic algorithm, and prioritizes the key nodes and core demand; makes the resource allocation operable by adjusting to form a resource scheduling strategy containing specific execution details, and avoids empty strategy; generates real-time control signals through rule conversion and encoding, ensures that the scheduling strategy can be effectively recognized and executed by the storage resource management module, and finally realizes the accurate, orderly and efficient dynamic allocation of storage resources in the system, solves the resource gap problem, avoids resource surplus waste, and improves the resource utilization rate and task execution adaptation degree. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0018] Figure 1 A flowchart of a storage resource dynamic allocation method based on artificial intelligence provided by an embodiment of the application;

[0019] Figure 2 A specific implementation diagram of a storage resource dynamic allocation method based on artificial intelligence provided by an embodiment of the application;

[0020] Figure 3A structural schematic diagram of a storage resource dynamic allocation system based on artificial intelligence is provided for an embodiment of the present application.

[0021] Figure 4 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0022] In scenarios such as cloud computing, big data processing, and AI training that require multiple storage nodes to cooperate, a target task usually goes through different stages such as data collection, calculation processing, and result output, and the demand for storage nodes is significantly different in each stage. For example, the data collection stage requires fast data transmission, and the calculation processing stage requires stronger calculation support. However, existing storage resource allocation techniques mostly rely on preset fixed templates, such as allocating resources evenly according to the number of nodes or adjusting resources only according to a single indicator such as the current transmission occupancy rate or calculation occupancy rate of the node, without considering the specific stage characteristics of the task. This approach can easily lead to resource "mismatch", for example, the data collection stage should be allocated transmission resources first, but a large amount of calculation resources are allocated; the calculation stage needs to allocate more calculation resources, but the transmission resource configuration is still high, which eventually leads to resource waste or slows down the task progress due to insufficient key resources.

[0023] To solve the problem that the resource allocation in the prior art is not matched with the demand of the task stage, the present application proposes a storage resource dynamic allocation method based on artificial intelligence. The core process of the method is as follows: first, the real-time response speed, data transmission speed, and calculation capability data of each storage node in the current stage of the target task are obtained; then these data are integrated and processed to determine the real-time load of each node; then an artificial intelligence model adapted to the current task stage is used to analyze these data to determine the resource allocation target that meets the stage demand; finally, a dynamic optimization algorithm is used to develop a specific resource scheduling scheme, and a control signal is generated to execute the resource allocation between nodes. This scheme uses "AI analysis adapted to the task stage" to avoid the limitations of the existing technology relying on general templates, ensuring that the resource allocation matches the demand of the current stage; through the integration and processing of multiple types of data, the one-sidedness of single indicator judgment is overcome, and the actual state of the node is accurately mastered; and finally, the resource allocation is adapted to the task stage and the node load, effectively solving the resource mismatch problem of the existing technology, improving the resource utilization rate, and ensuring efficient execution of the task.

[0024] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0025] The core of the present application is to provide an artificial intelligence-based storage resource dynamic allocation method, and a specific embodiment thereof has a flowchart as shown in the figure Figure 1 The method comprises the following steps. S101, acquiring real-time response delay data, real-time data transmission bandwidth data and real-time node computing power data of each storage node in the task stage of the target task.

[0026] In the above scheme, the target task refers to a business or technical task to be completed by multiple storage nodes, such as data processing, model training, etc.; the task stage refers to different processing links divided in the execution process of the target task, such as data import, calculation processing, result export, etc., and the performance requirements of different stages for storage nodes are different; the storage node refers to a hardware or logical unit used to carry storage resources and participate in data storage and interaction of the target task, such as an independent storage server in a distributed storage system, a logical storage unit in cloud storage, etc.; the real-time response delay data refers to the real-time time consumption data from receiving a data request to returning a response result of the storage node, which is used to reflect the response speed of the node; the real-time data transmission bandwidth data refers to the data transmission amount data that the storage node can actually complete in a unit of time, which is used to reflect the data transmission capacity of the node; the real-time node computing power data refers to the computing resource data of the storage node that can be used for data processing at present, which usually includes CPU idle rate, memory available amount, number of idle computing units, etc., and is used to reflect the computing support capacity of the node.

[0027] In the present application example, first, the target task and its current task stage are determined: the specific type of the target task to be executed is first determined, such as a distributed AI model training task, a large data batch processing task, etc., and then the current task stage of the target task is determined according to the preset execution process or real-time progress of the target task, the core requirement direction of the current stage for the performance of the storage node is determined, based on the result, all storage nodes participating in the current task stage are selected: according to the resource configuration list or system association record of the target task, the storage nodes having data interaction or resource support relationship with the current task stage are selected, the target node list to be collected is determined, and irrelevant node data interference is avoided.

[0028] Next, using the determined target storage node list, collect real-time response delay data of each node: adopt the "request-response" test method, send a predetermined number of test data requests to each target storage node, and record the time from each request to receive the response result, remove outliers, and take the average to obtain the real-time response delay data of each node. Based on the same target storage node list, collect real-time data transmission bandwidth data of each node: adopt the "real-time traffic monitoring method", record the total amount of received and transmitted data of the target node within a predetermined time period through the system built-in traffic monitoring tool, and then calculate the real-time data transmission bandwidth data according to the relationship "transmission bandwidth = total transmission data amount / monitoring time period". This data is used to reflect the current transmission capacity of the node and supplement the transmission dimension basis for performance analysis.

[0029] Finally, still based on the target storage node list, collect real-time node computing power data of each node: adopt the "resource occupation monitoring method", obtain the current CPU idle rate, memory available amount and other core computing resource parameters of the target node through the system resource monitoring tool, and then combine the preset computing power conversion rule such as "computing power = CPU total core number x CPU idle rate + memory available amount x preset computing power coefficient" to calculate the real-time node computing power data of each node. This data is used to reflect the current computing support capacity of the node and improve the computing dimension basis for performance analysis.

[0030] In actual application, a certain technology enterprise executes a distributed AI model training task, which is preset to be divided into three stages of data import, model training and result export. The task progress monitoring system determines that it is currently in the data import stage. According to the resource configuration list of the task, the storage nodes participating in the data import stage are selected as node A, node B and node C.

[0031] First, when collecting real-time response delay data of each node, 10 training data read requests of 200MB are sent to node A, B and C respectively, the time consumption of each request is recorded, outliers are removed, and the average value of all request time consumption after removing outliers is taken to obtain the real-time response delay of node A as 25 milliseconds, node B as 30 milliseconds and node C as 22 milliseconds. These data are used for subsequent analysis of the response speed adaptability of each node.

[0032] Then, when collecting the real-time data transmission bandwidth data of each node, the transmission data of each node within 5 minutes is recorded by a flow monitoring tool: node A receives 20 GB of data and sends 0.3 GB of data, node B receives 18 GB of data and sends 0.2 GB of data, and node C receives 22 GB of data and sends 0.4 GB of data. According to the formula real-time data transmission bandwidth = (received data volume + sent data volume) ÷ monitoring time, the received data volume and the sent data volume are in units of MB, and the monitoring time is in units of seconds; first, convert the data volume to MB (1 GB = 1024 MB), get the total transmission data volume of node A as 20 x 1024 + 0.3 x 1024 = 20787.2 MB, the monitoring time is 5 minutes = 300 seconds, and the real-time data transmission bandwidth of node A is 20787.2 ÷ 300 ≈ 69.3 MB / s; similarly, the real-time data transmission bandwidth of node B is (18 x 1024 + 0.2 x 1024) ÷ 300 ≈ 62.1 MB / s, and the real-time data transmission bandwidth of node C is (22 x 1024 + 0.4 x 1024) ÷ 300 ≈ 75.5 MB / s. These data are used for subsequent analysis of the transmission capacity matching degree of each node.

[0033] Then, when collecting the real-time node computing power data of each node, the parameters are obtained by a resource monitoring tool: the total number of CPU cores of node A is 32 cores, the CPU idle rate is 50%, and the available memory is 16 GB; the total number of CPU cores of node B is 32 cores, the CPU idle rate is 45%, and the available memory is 12 GB; the total number of CPU cores of node C is 32 cores, the CPU idle rate is 55%, and the available memory is 18 GB; according to the formula real-time node computing power = CPU total core number x CPU idle rate + memory available amount x memory computing power coefficient, the memory computing power coefficient is preset as 0.8, the real-time node computing power of node A is 32 x 50% + 16 x 0.8 = 16 + 12.8 = 28.8; the real-time node computing power of node B is 32 x 45% + 12 x 0.8 = 14.4 + 9.6 = 24; and the real-time node computing power of node C is 32 x 55% + 18 x 0.8 = 17.6 + 14.4 = 32. These real-time response delay data, real-time data transmission bandwidth data, and real-time node computing power data are all transmitted to the subsequent S102 step to generate a storage resource real-time performance data packet containing the real-time load state of each storage node.

[0034] The S101 overall scheme avoids collecting irrelevant nodes or data irrelevant to the current stage by first determining the target task and its current stage, and then reorienting and screening the storage nodes, ensuring the pertinence of data collection. At the same time, by using standardized methods to collect three types of real-time data reflecting the response speed, transmission capacity, and computing support capacity of the storage nodes, the core performance dimensions required for subsequent resource allocation analysis are fully covered, avoiding the one-sidedness of a single data dimension. The three types of data ultimately obtained can truly reflect the actual running state of each storage node in the current task stage, providing reliable and accurate basic input for the subsequent steps of data collaborative processing and AI model performance analysis, effectively ensuring the accuracy of subsequent storage resource allocation-related analysis, and laying a data foundation for realizing dynamic resource allocation that meets task requirements.

[0035] S102, the real-time response delay data, the real-time data transmission bandwidth data and the real-time node computing power data are processed cooperatively to obtain a storage resource real-time performance data packet containing real-time load states of each storage node.

[0036] Optionally, in step S102, the real-time response delay data, the real-time data transmission bandwidth data and the real-time node computing power data are processed cooperatively to obtain a storage resource real-time performance data packet containing real-time load states of each storage node, including: Step 1021, according to the real-time response delay data, the average delay and the delay jitter coefficient in a unit time window are calculated to form a delay feature set. Step 1022, the real-time data transmission bandwidth data is statistically processed in a sliding window to calculate the bandwidth utilization rate, the peak bandwidth proportion, the bandwidth fluctuation frequency and the effective transmission rate, and a bandwidth performance index set is generated. Step 1023, the real-time node computing power data is normalized, and the node CPU usage rate, the memory occupancy rate, the input / output operation number per second and the task queue length are combined to construct a computing power load evaluation value. Step 1024, using a three-dimensional collaborative analysis model, the delay feature set, the bandwidth performance index set and the computing power load evaluation value are spatio-temporally correlated and mapped to generate a comprehensive load index of each storage node, and the load state level is determined according to the comprehensive load index. Step 1025, the delay feature set, the bandwidth performance index set, the computing power load evaluation value, the comprehensive load index and the load state level of each storage node are structured and integrated to form a storage resource real-time performance data packet.

[0037] In the above scheme, the unit time window refers to a fixed time interval for statistical delay data; the average delay refers to the arithmetic mean of all real-time response delay data in the unit time window; the delay jitter coefficient refers to the ratio of the standard deviation of the delay data in the unit time window to the average delay; the delay feature set refers to the integration of the average delay and the delay jitter coefficient; the sliding window statistics refers to a method of collecting and statistically analyzing bandwidth data in a fixed time window; the bandwidth utilization rate refers to the ratio of real-time data transmission bandwidth to the maximum supported bandwidth of the node; the peak bandwidth ratio refers to the ratio of the peak bandwidth in a unit time to the maximum supported bandwidth of the node; the bandwidth fluctuation frequency refers to the number of times the bandwidth data exceeds the preset stable range in a unit time; the effective transmission refers to the ratio of the effective data transmission amount to the total transmission amount in a unit time; the bandwidth performance index set refers to the integration of the bandwidth utilization rate, the peak bandwidth ratio, the bandwidth fluctuation frequency and the effective transmission rate; the normalization processing refers to a processing method of converting computing power data of different ranges to a unified numerical range; the CPU usage rate refers to the ratio of the occupied CPU resources to the total CPU resources; the memory occupancy rate refers to the ratio of the occupied memory resources to the total memory resources; the input / output operations per second refers to the number of input / output operations that can be completed by the node in a unit time; the task queue length refers to the number of tasks currently waiting for processing of the node; the computing power load evaluation value refers to the integration of the normalized real-time node computing power data and the CPU usage rate, the memory occupancy rate, the IOPS and the task queue length; the three-dimensional collaborative analysis model refers to an analysis model for integrating delay, bandwidth and computing power three-dimensional data, which realizes collaborative analysis of multi-dimensional data through space-time correlation mapping; the comprehensive load index refers to a comprehensive numerical value reflecting the overall load degree of the node calculated by the three-dimensional collaborative analysis model; the load state level refers to the node load category divided based on the comprehensive load index; the storage resource real-time performance data packet refers to a data set containing complete performance and load information of the node formed by structurally integrating the delay feature set, the bandwidth performance index set, the computing power load evaluation value, the comprehensive load index and the load state level.

[0038] In the examples of the present application, firstly, the real-time response delay data obtained in S101 is processed in step 1021: firstly, a fixed unit time window is set, all real-time response delay data of a certain storage node in the window is aggregated, and the average delay is calculated according to the formula "average delay = sum of each delay data in the unit time window ÷ number of delay data collection in the unit time window", wherein the sum of each delay data in the unit time window refers to the cumulative value of all real-time response delays in the window, and the number of delay data collection in the unit time window refers to the total number of delay data collected in the window; then the delay jitter coefficient is calculated according to the formula "delay jitter coefficient = standard deviation of delay data in the unit time window ÷ average delay in the unit time window", wherein the standard deviation of delay data in the unit time window refers to the degree index of delay data deviating from the average delay in the window; and the two are integrated to form the delay feature set of the node, providing basic data of the delay dimension for subsequent multi-dimensional collaborative analysis.

[0039] Then, based on the processing logic of the delay data in step 1021, the real-time data transmission bandwidth data obtained in S101 is processed in step 1022: firstly, the sliding window length and the maximum supported bandwidth of the node are set, the bandwidth data is counted in segments in the sliding window unit, the bandwidth utilization rate is calculated according to the formula "bandwidth utilization rate = real-time data transmission bandwidth in the sliding window ÷ maximum supported bandwidth of the node", the peak bandwidth occupancy is calculated according to the formula "peak bandwidth occupancy = peak value of bandwidth in the sliding window ÷ maximum supported bandwidth of the node", wherein the peak value of bandwidth in the sliding window refers to the maximum value of real-time bandwidth in the window, the bandwidth fluctuation frequency is calculated according to the formula "bandwidth fluctuation frequency = number of times that the bandwidth exceeds the preset stable range in the sliding window", and the effective transmission rate is calculated according to the formula "effective transmission rate = effective data transmission amount in the sliding window ÷ total data transmission amount in the sliding window"; and the four indexes are integrated to form the bandwidth performance index set of the node, which together with the delay feature set constitutes two core dimension data for multi-dimensional analysis.

[0040] Then, the real-time node computing power data obtained in S101 is processed through step 1023: first, the minimum-maximum normalization method is used to convert the real-time node computing power data to a unified range of 0-1 according to the formula "normalized computing power data=(real-time node computing power original value-all node real-time computing power minimum value) ÷ (all node real-time computing power maximum value-all node real-time computing power minimum value)", wherein the real-time node computing power original value refers to the node computing power data obtained in S101, and the all node real-time computing power minimum value / maximum value refers to the minimum / maximum value in the original value of the computing power of all nodes participating in the current task stage; then the CPU usage ratio, i.e. the ratio of occupied CPU resources to total CPU resources, the memory occupancy ratio, i.e. the ratio of occupied memory resources to total memory resources, the input / output operation number per second and the task queue length of the node are obtained, and all four types of data are converted to the range of 0-1; according to the preset weight, i.e. the normalized computing power data 0.3, the CPU usage ratio 0.2, the memory occupancy ratio 0.2, the IOPS 0.15 and the task queue length 0.15, the sum of the weights is 1, and the computing power load evaluation value is calculated according to the formula "computing power load evaluation value=(normalized computing power data×0.3)+(CPU usage ratio×0.2)+(memory occupancy ratio×0.2)+(IOPS×0.15)+(task queue length×0.15)", the processing of the computing power dimension data is completed, and the third core dimension data for multi-dimensional analysis is formed.

[0041] Subsequently, multi-dimensional collaborative analysis is carried out through step 1024: a three-dimensional collaborative analysis model is called, the unit time window of step 1021, the sliding window of step 1022 and the collection time of the computing power data of step 1023 are aligned in the time dimension first to ensure that the three types of data reflect the node state of the same time period; then the delay feature set, the bandwidth performance index set and the computing power load evaluation value of the same node are associated in the space dimension, and the comprehensive load index is calculated according to the model preset dimension weight, i.e. the delay feature set 0.3, the bandwidth performance index set 0.3 and the computing power load evaluation value 0.4, according to the formula "comprehensive load index=(delay feature set score×0.3)+(bandwidth performance index set score×0.3)+(computing power load evaluation value×0.4)", wherein the delay feature set score and the bandwidth performance index set score are 0-1 scores converted by the model according to the specific values of the feature set and the index set; then the load state level of the node is determined according to the preset comprehensive load index range, i.e. 0-0.4 for low load, 0.4-0.7 for medium load and 0.7-1 for high load, to realize the quantification and classification of the overall load of the node.

[0042] Finally, all data is integrated through step 1025: the delay feature set of each node, the bandwidth performance index set, the computing power load evaluation value, the comprehensive load index, and the load state level are integrated according to the preset structured format of "node identification-delay feature set-bandwidth performance index set-computing power load evaluation value-comprehensive load index-load state level", forming a storage resource real-time performance data package containing the complete performance and load information of each node. This data package will be the core input for the performance analysis of the "artificial intelligence model corresponding to the task stage" in the subsequent S103 step, providing data support for obtaining the storage resource allocation target value.

[0043] In actual application, in the data import stage of distributed AI model training, for storage nodes A, B, and C participating in this stage, based on the real-time response delay data obtained in S101: node A is 25 ms, node B is 30 ms, node C is 22 ms, real-time data transmission bandwidth data: node A is 69.3 MB / s, node B is 62.1 MB / s, node C is 75.5 MB / s, real-time node computing power data: node A is 28.8, node B is 24, node C is 32, and collaborative processing is carried out.

[0044] First, when processing real-time response delay data, a 1-minute time window is set, and 12 delay data are collected in each node window, resulting in a sum of 25x12=300 ms for each delay data in the unit time window of node A, a collection frequency of 12, a sum of 30x12=360 ms for each delay data in the unit time window of node B, a collection frequency of 12, and a sum of 22x12=264 ms for each delay data in the unit time window of node C, a collection frequency of 12. According to the formula "average delay=sum of each delay data in the unit time window ÷ delay data collection frequency in the unit time window", the average delay of node A is 300÷12=25 ms, the average delay of node B is 360÷12=30 ms, and the average delay of node C is 264÷12=22 ms. According to the formula "delay jitter coefficient=standard deviation of delay data in the unit time window ÷ average delay in the unit time window", the delay jitter coefficient of each node is calculated, where the standard deviation of node A delay data is 2 ms, the jitter coefficient of node A is 2÷25=0.08, the standard deviation of node B delay data is 3 ms, the jitter coefficient of node B is 3÷30=0.1, the standard deviation of node C delay data is 1.5 ms, and the jitter coefficient of node C is 1.5÷22≈0.07. The delay feature set of each node is formed for subsequent collaborative analysis.

[0045] Then, when processing the real-time data transmission bandwidth data, a 10-second sliding window is set, the maximum supported bandwidth of each node is 100 MB / s, the total data transmission amount in the sliding window of each node is calculated according to the formula "total data transmission amount = real-time data transmission bandwidth x sliding window duration", wherein the total data transmission amount of node A = 69.3 MB / s x 10 s = 693 MB, the total data transmission amount of node B = 62.1 MB / s x 10 s = 621 MB, and the total data transmission amount of node C = 75.5 MB / s x 10 s = 755 MB; then the invalid data amount of each node is determined: because the transmission link stability of node A is moderate, the invalid data amount = 693 MB x 2% = 13.86 MB, the transmission link stability of node B is slightly poor, the invalid data amount = 621 MB x 3% = 18.63 MB, and the transmission link stability of node C is optimal, the invalid data amount = 755 MB x 1% = 7.55 MB; then the effective data transmission amount of each node is derived according to the formula "effective data transmission amount = total data transmission amount - invalid data amount", wherein the effective data transmission amount of node A = 693 MB - 13.86 MB = 679.14 MB, the effective data transmission amount of node B = 621 MB - 18.63 MB = 602.37 MB, and the effective data transmission amount of node C = 755 MB - 7.55 MB = 747.45 MB.

[0046] Then, the bandwidth utilization of each node is calculated according to the formula "bandwidth utilization = real-time data transmission bandwidth in the sliding window ÷ maximum supported bandwidth of the node", wherein the real-time data transmission bandwidth in the sliding window of node A is 69.3 MB / s, and thus the bandwidth utilization of node A is 69.3 ÷ 100 = 0.693, i.e. 69.3%, the real-time data transmission bandwidth in the sliding window of node B is 62.1 MB / s, and thus the bandwidth utilization of node B is 62.1 ÷ 100 = 0.621, i.e. 62.1%, and the real-time data transmission bandwidth in the sliding window of node C is 75.5 MB / s, and thus the bandwidth utilization of node C is 75.5 ÷ 100 = 0.755, i.e. 75.5%. The peak bandwidth ratio of each node is calculated according to the formula "peak bandwidth ratio = peak bandwidth in the sliding window ÷ maximum supported bandwidth of the node", wherein the peak bandwidth in the sliding window of node A is 72 MB / s, and thus the peak bandwidth ratio of node A is 72 ÷ 100 = 0.72, i.e. 72%, the peak bandwidth in the sliding window of node B is 65 MB / s, and thus the peak bandwidth ratio of node B is 65 ÷ 100 = 0.65, i.e. 65%, and the peak bandwidth in the sliding window of node C is 78 MB / s, and thus the peak bandwidth ratio of node C is 78 ÷ 100 = 0.78, i.e. 78%. The bandwidth fluctuation frequency of each node is counted according to the formula "bandwidth fluctuation frequency = number of times of bandwidth exceeding the preset stable range in the sliding window", wherein the preset stable range is 65-75 MB / s, the exceeding number of node A is 3, and thus the bandwidth fluctuation frequency of node A is 3, the exceeding number of node B is 2, and thus the bandwidth fluctuation frequency of node B is 2, the exceeding number of node C is 1, and thus the bandwidth fluctuation frequency of node C is 1. The effective transmission rate of each node is calculated according to the formula "effective transmission rate = effective data transmission amount in the sliding window ÷ total data transmission amount in the sliding window", and the effective transmission rate of node A is 679.14 MB ÷ 693 MB × 100% = 98%, the effective transmission rate of node B is 602.37 MB ÷ 621 MB × 100% = 97%, and the effective transmission rate of node C is 747.45 MB ÷ 755 MB × 100% = 99%. The bandwidth performance index set of the three nodes is integrated for subsequent collaborative analysis.

[0047] Afterwards, when processing the real-time node computing power data, first calculate the normalized computing power data of each node according to the formula "normalized computing power data = (real-time node computing power original value - all node real-time computing power minimum value) ÷ (all node real-time computing power maximum value - all node real-time computing power minimum value)", wherein the all node real-time computing power minimum value is 24 and the maximum value is 32, the real-time node computing power original value of node A is 28.8, the normalized computing power data of node A is (28.8-24) ÷ (32-24) = 0.6, the real-time node computing power original value of node B is 24, the normalized computing power data of node B is (24-24) ÷ (32-24) = 0, the real-time node computing power original value of node C is 32, the normalized computing power data of node C is (32-24) ÷ (32-24) = 1; then obtain the CPU usage of each node, that is, the CPU usage of node A is 0.5, the CPU usage of node B is 0.55, the CPU usage of node C is 0.45, obtain the memory occupancy rate of each node, that is, the memory occupancy rate of node A is 0.5, the memory occupancy rate of node B is 0.6, the memory occupancy rate of node C is 0.4, obtain the input / output operations per second (IOPS) of each node, that is, the input / output operations per second of node A is 0.7, the input / output operations per second of node B is 0.6, the input / output operations per second of node C is 0.8, obtain the task queue length of each node, that is, the task queue length of node A is 0.4, the task queue length of node B is 0.5, the task queue length of node C is 0.3, and calculate the computing power load evaluation value of each node according to the formula "computing power load evaluation value = (normalized computing power data × 0.3) + (CPU usage × 0.2) + (memory occupancy rate × 0.2) + (IOPS × 0.15) + (task queue length × 0.15)", the computing power load evaluation value of node A is (0.6 × 0.3) + (0.5 × 0.2) + (0.5 × 0.2) + (0.7 × 0.15) + (0.4 × 0.15) = 0.18 + 0.1 + 0.1 + 0.105 + 0.06 = 0.545, the computing power load evaluation value of node B is (0 × 0.3) + (0.55 × 0.2) + (0.6 × 0.2) + (0.6 × 0.15) + (0.5 × 0.15) = 0 + 0.11 + 0.12 + 0.09 + 0.075 = 0.395, the computing power load evaluation value of node C is (1 × 0.3) + (0.45 × 0.2) + (0.4 × 0.2) + (0.8 × 0.15) + (0.3 × 0.15) = 0.3 + 0.09 + 0.08 + 0.12 + 0.045 = 0.635, which is used for subsequent collaborative analysis.

[0048] Subsequently, a three-dimensional collaborative analysis model was invoked. Its time dimension aligns with the data from a 1-minute unit window and a 10-second sliding window, while its spatial dimension correlates the latency feature set, bandwidth performance index set, and computing load assessment value of each node. The comprehensive load index of each node was calculated using the formula: "Comprehensive Load Index = (Latency Feature Set Score × 0.3) + (Bandwidth Performance Index Set Score × 0.3) + (Computing Load Assessment Value × 0.4)". The latency feature set scores for node A were 0.6, for node B 0.65, and for node C 0.5; the bandwidth performance index set scores for node A were 0.65, for node B 0.6, and for node C 0.72. Therefore, the comprehensive load index for node A was (0.6 × 0.3) + (0 × 0.4) + (0 × 0.5) + (0 × 0.6 ... 0.65×0.3)+(0.545×0.4)=0.18+0.195+0.218=0.593; The comprehensive load index of node B is (0.65×0.3)+(0.6×0.3)+(0.395×0.4)=0.195+0.18+0.158=0.533; The comprehensive load index of node C is (0.5×0.3)+(0.72×0.3)+(0.635×0.4)=0.15+0.216+0.254=0.62; Based on the comprehensive load index range, namely 0-0.4 low load, 0.4-0.7 medium load, and 0.7-1 high load, nodes A, B, and C are all determined to be medium load level.

[0049] Finally, following the format of "node identifier - latency feature set - bandwidth performance index set - computing power load assessment value - comprehensive load index - load status level", all data of nodes A, B, and C are integrated to form a real-time performance data packet for storage resources. This data packet will be passed to the subsequent S103 step for performance analysis with the artificial intelligence model corresponding to the data import stage to obtain the target value for storage resource allocation for each node.

[0050] The aforementioned S102 overall solution, through targeted processing of three types of dispersed basic data, transforms single-dimensional latency, bandwidth, and computing power data into multi-dimensional data containing feature sets, indicator sets, and evaluation values, avoiding the one-sidedness of single data. Then, a three-dimensional collaborative analysis model is used to achieve spatiotemporal correlation of multi-dimensional data, generating a comprehensive load index and load status level that reflects the overall state of the nodes, making the node load situation more intuitive and comprehensive. Finally, all data is structured and integrated to form a real-time performance data package for storage resources. This not only eliminates the differences in magnitude and format between different data sets but also provides unified, complete, and accurate input information for the performance analysis of the artificial intelligence model in the subsequent S103 step, effectively ensuring the accuracy and efficiency of the subsequent analysis process. Simultaneously, it makes the performance and load status of storage nodes quantifiable and assessable, providing a scientific decision-making basis for the subsequent dynamic allocation of storage resources.

[0051] S103, performing performance analysis on the storage resource real-time performance data packet based on the artificial intelligence model corresponding to the task stage to obtain a storage resource allocation target value.

[0052] Optionally, step S103, performing performance analysis on the storage resource real-time performance data packet based on the artificial intelligence model corresponding to the task stage to obtain a storage resource allocation target value, comprises: Step 1031, obtaining a demand feature corresponding to the current task stage, the demand feature including a task data processing scale, a read-write operation frequency, a data persistence requirement, and a service quality level.

[0053] Step 1032, associating and integrating the storage resource real-time performance data packet with the demand feature to obtain to-be-analyzed data.

[0054] Step 1033, performing performance analysis on the to-be-analyzed data through the artificial intelligence model corresponding to the current task stage to obtain a performance target instruction of each storage node for meeting the demand of the current task stage.

[0055] Step 1033 can specifically include the following steps: through a comparison module in the artificial intelligence model corresponding to the current task stage, comparing the storage resource real-time performance data packet in the to-be-analyzed data with the demand feature to obtain difference information of each storage node in response speed, data transmission capacity, and processing capacity from the demand feature; through an analysis module in the artificial intelligence model, associating and analyzing the difference information with the real-time load state of the corresponding storage node to determine a shortage situation causing the difference; through an adjustment module in the artificial intelligence model, determining an adjustment direction of each storage node in response speed, data transmission capacity, and processing capacity for the shortage situation; through an integration module in the artificial intelligence model, integrating all the adjustment directions to form a performance target instruction of each storage node for meeting the demand of the current task stage.

[0056] Step 1034, based on a response speed adaptation requirement in the performance target instruction, combining the real-time response delay data with the real-time load state to obtain a response delay threshold, based on a data transmission capacity adaptation requirement in the performance target instruction, combining the real-time data transmission bandwidth data with the real-time load state to obtain a data transmission bandwidth quota, and based on a processing capacity adaptation requirement in the performance target instruction, combining the real-time node computing power data with the real-time load state to obtain a computing power support limit.

[0057] Step 1035, performing summary processing on all the response delay thresholds, the data transmission bandwidth quotas, and the computing power support limits to obtain a storage resource allocation target value of each storage node.

[0058] In the above scheme, the current task stage refers to the current processing link in the target task execution process, and the demand characteristics refer to the specific requirement parameter set of the current task stage for the storage resource, including the task data processing scale, the read-write operation frequency, the data persistence requirement, and the service quality level. The data to be analyzed is the data formed by integrating the storage resource real-time performance data packet and the demand characteristics, which is used to provide a unified analysis input for the artificial intelligence model. The artificial intelligence model corresponding to the current task stage is a special analysis model trained for the demand characteristics of the current stage, which includes four functional modules: comparison module, analysis module, adjustment module, and integration module. The comparison module is a submodule in the model for comparing the storage resource performance with the demand characteristics. The difference information refers to the gap data between the response speed, data transmission capacity, and processing capacity of each storage node and the demand characteristics. The analysis module is a submodule in the model for associating the difference information with the node load state. The shortage situation refers to the specific resource deficiency type that causes the difference between the node performance and the demand. The adjustment module is a submodule in the model for determining the performance optimization direction of the node. The adjustment direction refers to the performance dimension that the node needs to improve or optimize according to the shortage situation. The integration module is a submodule in the model for summarizing the adjustment direction. The performance target instruction is a structured instruction that includes the adjustment direction of each node. The response delay threshold is the maximum response delay standard that the node needs to reach based on the response speed demand in the performance target instruction and the real-time load state of the node. The data transmission bandwidth quota is the minimum bandwidth quota that the node can allocate based on the transmission capacity demand in the performance target instruction and the real-time load state of the node. The computing power support quota is the minimum computing power standard that the node needs to reach based on the processing capacity demand in the performance target instruction and the real-time computing power data and load state of the node. The storage resource allocation target value is the resource allocation benchmark value that each node needs to meet after summarizing the response delay threshold, data transmission bandwidth quota, and computing power support quota of all nodes, which is used to develop the subsequent scheduling strategy.

[0059] In the present application example, first, the demand characteristics of the current task stage are obtained through the 1031 step: first, according to the stage definition of the target task, such as the data import stage and the computing processing stage of the distributed AI model training, the core resource demand direction of the stage is determined, such as high bandwidth and low delay in the data import stage. Then, the specific demand parameters of the stage are extracted through the task management system, including the task data processing scale, i.e., the total amount of data to be processed in the current stage, the read-write operation frequency, i.e., the number of read-write requests to the storage node per unit time, the data persistence requirement, i.e., the attribute of long-term storage or temporary storage of data, and the service quality level, i.e., the minimum standard of response delay and transmission bandwidth. These parameters are integrated to form the demand characteristics, which will be used as the benchmark for subsequent judgment of whether the node performance is suitable for the task demand.

[0060] Then, based on the demand characteristics obtained in step 1031, the data to be analyzed is generated by step 1032: first, the key performance indicators of each node in the storage resource real-time performance data package output in step S102 are extracted, such as response delay, data transmission bandwidth, and computing power load evaluation value, and then these performance indicators are bound one by one with the corresponding parameters in the demand characteristics, such as the response delay of the node binding the delay requirement of the service quality level, the bandwidth of the node binding the bandwidth requirement of the service quality level, forming the data to be analyzed containing the corresponding relationship of "node actual performance-task demand standard", which will be used as the analysis input of the artificial intelligence model corresponding to the current task stage, ensuring that the model can analyze the node state and task demand at the same time.

[0061] Then, taking the data to be analyzed generated in step 1032 as input, the performance target instruction is generated by step 1033 using the artificial intelligence model: first, the comparison module of the model is called to compare each node's performance indicators in the data to be analyzed with the corresponding standards in the demand characteristics one by one, output the difference information, if the performance meets the standard, mark "no difference", if not, quantify the difference; second, the analysis module of the model is called to associate the difference information with the real-time load state of the corresponding node, i.e. the load state level output in step S102, such as low load, medium load, high load, if there is a difference, determine the shortage situation that leads to the difference, such as high load leading to excessive response delay, insufficient bandwidth leading to substandard transmission capacity, if there is no difference, determine "no shortage"; third, the adjustment module of the model is called to determine the performance adjustment direction of each node for the shortage situation, such as "insufficient bandwidth" for the adjustment direction "improve bandwidth quota", and no shortage for the adjustment direction "maintain current performance and optimize stability"; fourth, the integration module of the model is called to summarize the adjustment direction of each node in the format of "node identification-adjustment direction", forming the performance target instruction, which will be used to guide the calculation of the subsequent resource performance target parameters.

[0062] Then, based on the performance target instruction generated in step 1033, the resource performance target parameters of each node are calculated through step 1034: first, for the response speed adaptation requirement in the performance target instruction, combined with the real-time response delay data obtained in step S101 and the real-time load state in step S102, the response delay threshold is calculated according to the formula: response delay threshold = real-time response delay data x load influence coefficient. The load influence coefficient is set according to the node load state, for example, higher coefficient for high load, reserving more performance redundancy to respond to load fluctuations; second, for the data transmission capacity adaptation requirement in the performance target instruction, combined with the node real-time data transmission bandwidth data in step S101 and the real-time load state in step S102, the data transmission bandwidth quota is calculated according to the formula: data transmission bandwidth quota = real-time data transmission bandwidth data x load influence coefficient; third, for the processing capacity adaptation requirement in the performance target instruction, combined with the node real-time node computing power data in step S101 and the real-time load state in step S102, the computing power support quota is calculated according to the formula: computing power support quota = real-time node computing power data x load influence coefficient. The three parameters together constitute the resource performance target that the node needs to meet.

[0063] Finally, the storage resource allocation target value is generated through step 1035: first, according to the "node identification" classification, the response delay threshold, data transmission bandwidth quota and computing power support quota of each node are collected; then, according to the demand characteristics obtained in step 1031, check whether the parameters meet the bottom line requirements of the quality of service level, for example, whether the response delay threshold is ≤ the delay upper limit in the demand, whether the bandwidth quota is ≥ the bandwidth lower limit in the demand, if not, adjust to meet the requirements; finally, the three types of parameters of each node are integrated into structured data to form the storage resource allocation target value of each node, which will be the core basis for formulating resource scheduling strategies through dynamic allocation optimization algorithm in subsequent step S104.

[0064] In actual application, in the data import stage of distributed AI model training, the storage nodes A, B and C participating in this stage are processed through step S103. First, the demand characteristics of this stage are obtained: the task data processing scale is 100GB, the read-write operation frequency is 500 data write requests per second, the data persistence requirement is temporary storage, the data does not need to be retained for a long time, and the quality of service level is response delay ≤ 30 milliseconds and data transmission bandwidth ≥ 60MB / s. These demand characteristics will be used as the basis for judging the performance adaptation of the nodes.

[0065] Then, the storage resource real-time performance data packet output in step S102 is associated, which contains the average delay of node A 25 ms, data transmission bandwidth 69.3 MB / s, and computing power load evaluation value 0.545; the average delay of node B 30 ms, data transmission bandwidth 62.1 MB / s, and computing power load evaluation value 0.395; and the average delay of node C 22 ms, data transmission bandwidth 75.5 MB / s, and computing power load evaluation value 0.635. The performance indicators of each node are bound to the corresponding demand characteristics to form data to be analyzed, for example, the data to be analyzed of node B contains the corresponding relationship of “average delay 30 ms-demand response delay ≤ 30 ms, data transmission bandwidth 62.1 MB / s-demand bandwidth ≥ 60 MB / s”, which will be used for performance analysis of the subsequent artificial intelligence model.

[0066] Then, the artificial intelligence model corresponding to the data import stage is called, the first step is to analyze the data to be analyzed by the comparison module, node A average delay 25 ms ≤ 30 ms, bandwidth 69.3 MB / s ≥ 60 MB / s, node B average delay 30 ms = 30 ms, bandwidth 62.1 MB / s ≥ 60 MB / s, node C average delay 22 ms ≤ 30 ms, bandwidth 75.5 MB / s ≥ 60 MB / s, all node performances meet the demand characteristics, and the difference information output is “all node response speed and data transmission capacity meet the demand characteristics”; the second step is to determine the load state of each node by the analysis module in step S102, and determine that the shortage condition is “no resource shortage”; the third step is to determine the adjustment direction of each node by the adjustment module, which is “maintain the current performance level and optimize the data transmission stability”; and the fourth step is to integrate the adjustment direction by the integration module to form the performance target instruction: “node A: maintain the current response delay and bandwidth, and optimize the transmission stability; node B: maintain the current response delay and bandwidth, and optimize the transmission stability; node C: maintain the current response delay and bandwidth, and optimize the transmission stability”, which will be used to calculate the resource performance target parameters of each node.

[0067] After that, based on the performance target instruction, combined with the real-time data obtained in step S101: node A real-time response delay 25 milliseconds, bandwidth 69.3 MB / s, computing power 28.8, node B real-time response delay 30 milliseconds, bandwidth 62.1 MB / s, computing power 24, node C real-time response delay 22 milliseconds, bandwidth 75.5 MB / s, computing power 32, combined with the medium load state in step S102, the load influence coefficient corresponding to the medium load is set to 1.05, wherein the load influence coefficient is set according to the load level, and a higher coefficient is required for high load to reserve redundancy, and the medium load is set to 1.05 to balance the performance and redundancy demand. According to the formula "response delay threshold = real-time response delay data x load influence coefficient", the response delay threshold of each node is calculated, and the response delay threshold of node A is 25x1.05=26.25 milliseconds, the response delay threshold of node B is 30x1.05=31.5 milliseconds, because 31.5 milliseconds>30 milliseconds demand bottom line, so the response delay threshold of node B is adjusted to 30 milliseconds, and the response delay threshold of node C is 22x1.05=23.1 milliseconds; According to the formula "data transmission bandwidth quota = real-time data transmission bandwidth data x load influence coefficient", the data transmission bandwidth quota of each node is calculated, and the data transmission bandwidth quota of node A is 69.3x1.05≈72.77 MB / s, the data transmission bandwidth quota of node B is 62.1x1.05≈65.21 MB / s, and the data transmission bandwidth quota of node C is 75.5x1.05≈79.28 MB / s; According to the formula "computing power support limit = real-time node computing power data x load influence coefficient", the computing power support limit of each node is calculated, and the computing power support limit of node A is 28.8x1.05=30.24, the computing power support limit of node B is 24x1.05=25.2, and the computing power support limit of node C is 32x1.05=33.6.

[0068] Finally, the response delay threshold, data transmission bandwidth quota and computing power support limit of each node are summarized to form the storage resource allocation target value of each node, and the storage resource allocation target value of node A is 26.25 milliseconds, 72.77 MB / s and 30.24; the storage resource allocation target value of node B is 30 milliseconds, 65.21 MB / s and 25.2; and the storage resource allocation target value of node C is 23.1 milliseconds, 79.28 MB / s and 33.6. The storage resource allocation target value will be passed to subsequent step S104 to formulate the resource scheduling strategy of each storage node by using a dynamic allocation optimization algorithm.

[0069] The above S103 overall scheme, by first clearly defining the demand characteristics of the current task stage, the resource analysis has a clear goal orientation, avoiding blind analysis that is divorced from task demand; then the demand characteristics and node performance data are integrated into the data to be analyzed, ensuring that the analysis input of the artificial intelligence model contains both the actual state of the node and the task demand, improving the analysis pertinence; through the special artificial intelligence model with multiple module functions, the differences between node performance and demand are accurately identified, the difference reasons are determined and the adjustment direction is determined, avoiding the subjectivity and one-sidedness of manual analysis; based on the adjustment direction, the response delay threshold, bandwidth quota and computing power quota are calculated according to the node load state, so that the resource target not only meets the demand bottom line, but also considers the load fluctuation to reserve redundancy, ensuring the feasibility of the target; finally, the storage resource allocation target value formed by the final summary provides a quantitative and accurate benchmark for subsequent resource scheduling strategy, effectively ensuring that the subsequent resource allocation can meet the current task stage demand and node actual state, avoiding resource mismatch or performance not meeting the standard, and laying a scientific target foundation for dynamic allocation of storage resources.

[0070] S104, based on the storage resource allocation target value, a resource scheduling strategy is formulated by a dynamic allocation optimization algorithm, and a real-time control signal corresponding to the resource scheduling strategy is generated, and based on the real-time control signal, the dynamic allocation and scheduling of storage resources between each storage node are executed.

[0071] Optionally, step S104, based on the storage resource allocation target value, a resource scheduling strategy is formulated by a dynamic allocation optimization algorithm, and a real-time control signal corresponding to the resource scheduling strategy is generated, and based on the real-time control signal, the dynamic allocation and scheduling of storage resources between each storage node are executed. Step 1041, according to the storage resource allocation target value, the hardware specifications and task load characteristics of each storage node are combined to calculate the resource demand list of each storage node in the current task stage, and the resource demand list includes: storage capacity, read-write throughput, storage resource type and quantity.

[0072] Step 1042, collect real-time available resource data of all storage nodes, and form a global available resource list, compare the resource demand list of each storage node with the global available resource list item by item, and according to the comparison result, quantitatively statistics the supply-demand difference of each type of resource, wherein the part exceeding the supply is the resource gap, and the part exceeding the demand is the resource surplus, wherein the global available resource list represents the total amount of actual available resources of all storage nodes.

[0073] Step 1043, based on the supply-demand difference of each type of resource, the resource allocation sequence of each storage node is determined by a dynamic allocation optimization algorithm.

[0074] Step 1043 specifically includes the following steps: Based on the supply and demand difference of various resources, determine the corresponding allocation gap or surplus information, wherein the allocation gap or surplus information includes: resource gap or surplus type, scale, and scope of impact; classify the allocation gap or surplus information according to the type of response latency resources, data transmission bandwidth resources, and computing power resources to obtain the gap degree or surplus degree corresponding to each type of resource; compare the gap degree or surplus degree of each type of resource with the corresponding preset resource importance level to determine the allocation priority of the corresponding resource in the current task stage; extract the demand value of each type of resource in each storage node and the urgency level identifier of the current task carried by each storage node from the resource demand list; associate the demand value with the urgency level identifier, and based on the association result, sort each storage node under the same resource according to the preset sorting rules to obtain the node allocation sub-order corresponding to the resource; and integrate all the node allocation sub-orders in series according to the allocation priority to obtain the resource allocation order of each storage node.

[0075] Step 1044: Adjust the resource requirement list of each storage node according to the resource allocation order to form a resource scheduling strategy for each storage node. The resource scheduling strategy includes: resource allocation path, adjustment range and implementation time.

[0076] Step 1044 specifically includes the following steps: Following the resource allocation order, select the resource type with the highest current priority to obtain the resource to be processed; according to the node allocation sub-order corresponding to the resource to be processed, select the current processing node sequentially; obtain the current processing node's demand value for the resource to be processed, the total supply of the resource to be processed, and the cumulative amount already allocated to other nodes; calculate the difference between the total supply and the cumulative amount to obtain the remaining allocable amount; if the demand value is less than or equal to the remaining allocable amount, then use the demand value as the actual allocation amount for the current processing node; or, if the demand value is greater than the remaining allocable amount, then, combined with the real-time load status of the current processing node and the upper limit of the carrying capacity of the resource to be processed, use a value that does not exceed the remaining allocable amount and does not exceed the upper limit of the carrying capacity as the actual allocation amount for the current processing node; record the actual allocation amount as the final allocation result of the current processing node for the resource to be processed; integrate the final allocation results of each storage node in various types of resources to form a resource scheduling strategy for each storage node.

[0077] Step 1045: Perform rule conversion on the resource scheduling strategy to obtain instruction parameters, encode the instruction parameters according to a preset signal format, and generate a real-time control signal corresponding to the resource scheduling strategy.

[0078] In the above scheme, the hardware specification refers to the hardware configuration parameters of each storage node; the task load characteristic refers to the load characteristics of the node in the current task stage; the resource requirement list refers to the node resource requirement details formed by integrating storage capacity, read-write throughput, storage resource type and quantity; the storage capacity refers to the total amount of data that the node needs to carry; the read-write throughput refers to the amount of read-write data that the node needs to complete per unit time; the real-time available resource data refers to the resource data currently not occupied by each node; the global available resource list refers to the system-level resource list formed by aggregating the real-time available resource data of all nodes; the supply-demand gap refers to the resource supply-demand difference obtained by comparing the resource requirement list with the global available resource list, the part exceeding the supply is the resource gap, and the part exceeding the demand is the resource surplus; the dynamic allocation optimization algorithm refers to the algorithm for formulating the resource allocation order and optimizing the resource allocation efficiency; the preset resource importance level refers to the resource priority preset according to the task stage requirement; the emergency level identifier refers to the label marking the current task emergency level of each node; the node allocation sub-sequence refers to the node allocation sequence obtained according to the ordering rule under the same resource type; the resource allocation sequence refers to the system-level resource allocation process formed by concatenating all node allocation sub-sequences according to the resource priority; the resource scheduling strategy refers to the node resource allocation scheme including the resource allocation path, the adjustment range and the implementation time; the resource allocation path refers to the transmission path of the resource from the surplus node to the gap node; the adjustment range refers to the specific value of the resource allocation or transfer; the implementation time refers to the execution time point of the resource allocation; the instruction parameter refers to the machine recognizable parameter converted from the resource scheduling strategy; the preset signal format refers to the control signal coding format agreed by the system; the real-time control signal refers to the execution signal generated by encoding the instruction parameter according to the preset signal format, which is used to trigger each node to perform the resource allocation operation.

[0079] In practical application, in the data import stage of distributed AI model training, S104 step processing is carried out for storage nodes A, B and C, and the output results of S101, S102 and S103 need to be associated, that is, S103 has determined the storage resource allocation target value of each node: node A is 72.77 MB / s bandwidth, 30.24 algorithm, node B is 65.21 MB / s bandwidth, 25.2 algorithm, and node C is 79.28 MB / s bandwidth, 33.6 algorithm; S102 has determined that each node is in a medium load state; S101 has collected the basic real-time data of each node, including the real-time response delay of node A is 25 milliseconds, the bandwidth is 69.3 MB / s, the algorithm is 28.8, the real-time response delay of node B is 30 milliseconds, the bandwidth is 62.1 MB / s, the algorithm is 24, and the real-time response delay of node C is 22 milliseconds, the bandwidth is 75.5 MB / s, and the algorithm is 32.

[0080] First, the resource requirement list is calculated: the hardware specifications of each node are set to be consistent, with a maximum supported bandwidth of 100 MB / s, a maximum storage capacity of 500 GB, and a CPU of 32 cores; the current task data processing scale is 100 GB, and the data volume is allocated according to the node carrying capacity, i.e., node A bears 40%, node B bears 35%, and node C bears 25%; the storage capacity of each node is calculated according to the formula "storage capacity = task data processing scale x node carrying ratio", and the storage capacity of node A is 40 GB, the storage capacity of node B is 35 GB, and the storage capacity of node C is 25 GB; according to the rule that the read-write throughput is consistent with the bandwidth quota, the read-write throughput of node A is 72.77 MB / s, the read-write throughput of node B is 65.21 MB / s, and the read-write throughput of node C is 79.28 MB / s; the storage resource quantity corresponds to the target value determined by S103, i.e., the bandwidth of node A is 72.77 MB / s, the computing power requirement of node A is 30.24, the bandwidth of node B is 65.21 MB / s, the computing power requirement of node B is 25.2, the bandwidth of node C is 79.28 MB / s, and the computing power requirement of node C is 33.6; the above data is integrated to form a resource requirement list of each node, which is used for supply and demand comparison in the future.

[0081] Next, the supply and demand difference is calculated: the real-time available resource data of each node is collected, and the available bandwidth of node A is 25 MB / s, the available computing power is 8, and the available storage is 100 GB; the available bandwidth of node B is 30 MB / s, the available computing power is 10, and the available storage is 80 GB; the available bandwidth of node C is 20 MB / s, the available computing power is 12, and the available storage is 120 GB; the available bandwidth, available computing power, and available storage of each node are summarized to form a global available resource list, and the global available bandwidth is 75 MB / s, the global available computing power is 30, and the global available storage is 300 GB; the supply and demand difference is calculated, the global bandwidth demand is 217.26 MB / s, the bandwidth gap is 142.26 MB / s, the global computing power demand is 89.04, the computing power gap is 59.04, and the global storage demand is 100 GB; the storage surplus is 200 GB; the supply and demand difference is used to determine the resource allocation order in the future.

[0082] Then, the resource allocation order is determined: the preset resource importance level of the data import stage is set as "bandwidth resource>computing power resource>storage resource", the allocation gap or surplus information is generated based on the supply-demand gap, it is obtained that the bandwidth is a heavy gap, the computing power is a heavy gap, and the storage is a light surplus, then the resource demand values of each node are extracted, it is obtained that the bandwidth demand of node C is 79.28>the bandwidth demand of node A is 72.77>the bandwidth demand of node B is 65.21, the computing power demand of node C is 33.6>the computing power demand of node A is 30.24>the computing power demand of node B is 25.2, the storage demand of node A is 40>the storage demand of node B is 35>the storage demand of node C is 25, the task urgency of each node is medium, according to the "demand value large priority" ordering rule, the node allocation sub-order is obtained, the bandwidth is C→A→B, the computing power is C→A→B, and the storage is A→B→C; according to the resource priority series sub-order, the resource allocation order is formed, the bandwidth is allocated according to C→A→B first, then the computing power is allocated according to C→A→B, and finally the storage is allocated according to A→B→C; the allocation order is used subsequently to develop a resource scheduling strategy.

[0083] Afterwards, the resource scheduling strategy is formed: the bandwidth resources are processed in the order of resource allocation, and the first processing node is C. The remaining allocatable amount of each node is calculated according to the formula "remaining allocatable amount = total supply amount - allocated cumulative amount", wherein the total supply amount is the globally available bandwidth 75 MB / s, and the allocated cumulative amount is 0 for the first allocation. The remaining allocatable amount of node C = 75 - 0 = 75 MB / s. Since the bandwidth demand of node C is 79.28 MB / s, which is greater than 75 MB / s, and the corresponding bandwidth bearing upper limit of the load is 80 MB / s, the actual allocation amount of node C is 75 MB / s, and the allocated cumulative amount is updated to 75 MB / s. Since the bandwidth demand of node A is 72.77 MB / s and the bandwidth demand of node B is 65.21 MB / s, which exceeds the remaining allocatable amount, 72.77 MB / s and 65.21 MB / s are planned to be allocated from the reserved resource pool, respectively. Then the computing power resources are processed, and the first processing node is C. The total supply amount is the globally available computing power 30, and the allocated cumulative amount is 0. The remaining allocatable amount = 30 - 0 = 30. Since the computing power demand of node C is 33.6, which is greater than 30, and the corresponding computing power bearing upper limit of the load is 32, the actual allocation amount of node C is 12, and the allocated cumulative amount is updated to 12. Similarly, node A is allocated available computing power 8, and the allocated cumulative amount is updated to 20. Node B is allocated available computing power 10, and the allocated cumulative amount is updated to 30. The computing power gap of node A = 30.24 - 8 = 22.24, the computing power gap of node B = 25.2 - 10 = 15.2, and the computing power gap of node C = 33.6 - 12 = 21.6. It is planned to supplement from the reserved resource pool. Then the storage resources are processed, and the storage capacity of node A is 40 GB, the storage capacity of node B is 35 GB, and the storage capacity of node C is 25 GB, all of which are within the globally available storage range. The above data are integrated to form the resource scheduling strategy of each node, i.e. node A is "allocation path: reserved resource pool -> A; adjustment range: bandwidth + 72.77 MB / s, computing power + 22.24, storage + 40 GB; implementation time: T + 10 seconds", node B is "allocation path: reserved resource pool -> B; adjustment range: bandwidth + 65.21 MB / s, computing power + 15.2, storage + 35 GB; implementation time: T + 10 seconds", and node C is "allocation path: reserved resource pool -> C; adjustment range: bandwidth + 4.28 MB / s, computing power + 21.6, storage + 25 GB; implementation time: T + 10 seconds". The scheduling strategy is used for subsequent conversion to real-time control signals.

[0084] Finally, the real-time control signal is generated: the node resource scheduling strategy is converted into instruction parameters, and the instruction parameters of node A are "allocation path: Pool→A; bandwidth adjustment value: 72.77; computing power adjustment value: 22.24; storage adjustment value: 40; implementation time: T+10", the instruction parameters of node B are "allocation path: Pool→B; bandwidth adjustment value: 65.21; computing power adjustment value: 15.2; storage adjustment value: 35; implementation time: T+10", and the instruction parameters of node C are "allocation path: Pool→C; bandwidth adjustment value: 4.28; computing power adjustment value: 21.6; storage adjustment value: 25; implementation time: T+10"; the instruction parameters of each node are encoded according to the preset JSON format, and the encoding of node A is "{"node":"A","path":"Pool→A","bandwidth":72.77,"computing":22.24,"storage":40,"time":"T+10"}", the encoding of node B is "{"node":"B","path":"Pool→B","bandwidth":65.21,"computing":15.2,"storage":35,"time":"T+10"}", and the encoding of node C is "{"node":"C","path":"Pool→C","bandwidth":4.28,"computing":21.6,"storage":25,"time":"T+10"}"; and a real-time control signal corresponding to the scheduling strategy is generated, and the signal is sent to each storage node and the reserved resource pool. The signal will trigger the dynamic allocation and scheduling of storage resources in each node in the future, and complete the connection with the subsequent execution link.

[0085] The above S104 overall scheme combines the storage resource allocation target value with the node hardware specifications and the task load characteristics to calculate the resource demand list, clearly specifies the specific types and scales of resource demands of each node, and avoids ambiguous resource allocation directions; by collecting real-time available resources and summarizing the global list, and then comparing and calculating the supply-demand gap with the demand list, the system resource supply and demand status is clearly mastered, providing objective data support for resource allocation decisions; based on the dynamic allocation optimization algorithm, the allocation sequence is determined in combination with the resource gap degree and the task stage priority, ensuring that the resource priority meets the key demand and improving the pertinence of resource allocation; the demand list is adjusted according to the allocation sequence, and the allocation path, adjustment range and implementation time are specified, so that the resource scheduling strategy has executable execution details and avoids empty strategies; the real-time control signal is generated through rule conversion and coding, ensuring that the scheduling instruction can be accurately recognized and triggered by the system to realize the closed loop from the target value to the actual resource allocation.

[0086] The following is a complete example for steps S101-S104, as shown in the following table: Figure 2 As shown in the following table, first, 10 times of 200MB training data read requests are sent to each of the three nodes, the time consumption of each time is recorded, and the real-time response delay data is calculated: the time consumption of node A for 10 times is 22ms, 23ms, 22ms, 24ms, 23ms, 22ms, 23ms, 24ms, 22ms, and 23ms, respectively, the average delay of each node is calculated according to the formula "average delay = sum of each delay data in a unit time window ÷ number of delay data collection in a unit time window", and the average delay of node A is 22.8ms, and the average delay of node B is 30.2ms, and the average delay of node C is 21.5ms; second, the data transmission of each node within 1 minute is recorded by a flow monitoring tool, and the real-time data transmission bandwidth data is calculated: the received data of node A is 3.8GB, and the sent state data is 0.2GB, the data amount is converted to MB (1GB = 1024MB), and the total transmission amount is 4096MB, the bandwidth of each node is calculated according to the formula "real-time data transmission bandwidth = total transmission amount ÷ monitoring time", and the bandwidth of node A is 68.27MB / s, and the bandwidth of node B is 61.8MB / s, and the bandwidth of node C is 74.9MB / s; third, the hardware parameters of each node are obtained and the real-time node computing power data is calculated, wherein the total CPU core number of node A is 32 cores, the CPU idle rate is 60%, and the memory available amount is 18GB, the real-time node computing power of each node is calculated according to the formula "real-time node computing power = CPU total core number × CPU idle rate + memory available amount × memory computing power coefficient", and the memory computing power coefficient is preset to 0.8, the real-time node computing power of node A is 33.6, and the real-time node computing power of node B is 24.5, and the real-time node computing power of node C is 35.2.

[0087] Then, the first step is to process the real-time response delay data to generate the delay feature set based on the obtained three types of real-time data: the node A delay data standard deviation is 1.2 milliseconds, the delay jitter coefficient of each node is calculated according to the formula "delay jitter coefficient = standard deviation of delay data in unit time window ÷ average delay in unit time window", the delay jitter coefficient of node A is 1.2 ÷ 22.8 ≈ 0.053, and the delay jitter coefficients of nodes B and C are 0.08 and 0.04 respectively, and the average delay and jitter coefficient are integrated to form the delay feature set of each node; the second step is to process the real-time data transmission bandwidth data to generate the bandwidth performance index set: the maximum supported bandwidth of node A is 100 MB / s, the bandwidth utilization rate of each node is calculated according to the formula "bandwidth utilization rate = real-time data transmission bandwidth in sliding window ÷ node maximum supported bandwidth", the bandwidth utilization rate of node A is 68.27 ÷ 100 = 0.6827, i.e. 68.27%, the peak bandwidth occupancy of each node is calculated according to the formula "peak bandwidth occupancy = peak bandwidth in sliding window ÷ node maximum supported bandwidth", the peak bandwidth occupancy of node A is 71%, the bandwidth fluctuation frequency of node A is 2 times, the effective data transmission rate is 98%, and the bandwidth utilization rate, peak bandwidth occupancy, fluctuation frequency, and effective transmission rate of nodes B and C are calculated in the same way to form the bandwidth performance index set of each node; the third step is to process the real-time node computing power data to generate the computing power load evaluation value: first, normalize all node computing power according to the formula "normalized computing power data = (real-time node computing power original value - all node real-time computing power minimum value) ÷ (all node real-time computing power maximum value - all node real-time computing power minimum value)", where the minimum value of all node computing power is 24.5 and the maximum value is 35.2, the normalized computing power of node A is (33.6 - 24.5) ÷ (35.2 - 24.5) ≈ 0.85, then obtain the CPU usage rate of each node, the CPU usage rate of node A is 40%, the CPU usage rate of node B is 45%, the CPU usage rate of node C is 35%, the memory occupancy rate of each node is obtained, the memory occupancy rate of node A is 45%, the memory occupancy rate of node B is 50%, the memory occupancy rate of node C is 40%, the IOPS of each node is obtained, the IOPS of node A is 0.75, the IOPS of node B is 0.65, the IOPS of node C is 0.8, the task queue length of each node is obtained, the task queue length of node A is 0.35, the task queue length of node B is 0.4, and the task queue length of node C is 0.3, and the computing power load evaluation value is calculated according to the formula "computing power load evaluation value = normalized computing power data × 0.3 + CPU usage rate × 0.2 + memory occupancy rate × 0.2+IOPSx0.15+task queue lengthx0.15” to obtain the computing power load evaluation value of node A = 0.85x0.3+0.4x0.2+0.45x0.2+0.75x0.15+0.35x0.15 = 0.255+0.08+0.09+0.1125+0.0525 = 0.59, and the evaluation value of node B is 0.62 and the evaluation value of node C is 0.55; the fourth step is to call a three-dimensional collaborative analysis model to generate a comprehensive load index and a load state level: according to the formula “comprehensive load index = delay characteristic set scorex0.3+bandwidth performance index set scorex0.3+computing power load evaluation valuex0.4” to calculate the comprehensive load index of each node, wherein the delay characteristic set score of node A is 0.55 and the bandwidth performance index set score is 0.62, to obtain the comprehensive load index of node A = 0.55x0.3+0.62x0.3+0.59x0.4 = 0.165+0.186+0.236 = 0.587, according to the range of the comprehensive load index, i.e. 0-0.4 low load, 0.4-0.7 medium load, and 0.7-1 high load, it is determined that node A is medium load, and nodes B and C are medium load; the fifth step is to integrate the delay characteristic set, the bandwidth performance index set, the computing power load evaluation value, the comprehensive load index, and the load state level to form a storage resource real-time performance data package, which will be used for AI model performance analysis in the third step.

[0088] Then, the first step determines the demand characteristics of the current task stage: the task data processing scale is 100 GB, the read-write operation frequency is 500 times / s, the data persistence requirement is temporary storage, the service quality level response delay is ≤30 ms, and the data transmission bandwidth is ≥60 MB / s; the second step associates the generated storage resource real-time performance data packet with the demand characteristics to form the data to be analyzed: the average delay of each node is bound with the delay requirement of the service quality level, and the bandwidth utilization is bound with the bandwidth requirement to ensure that the data to be analyzed contains the "node performance-demand standard" correspondence; the third step calls the artificial intelligence model corresponding to the data import stage to process the data to be analyzed: the model comparison module compares the node performance and demand standard item by item, determines that the delay of node A is 22.8 ms≤30 ms, and the bandwidth is 68.27 MB / s≥60 MB / s, and there is no difference, and similarly, it is determined that nodes B and C have no difference, and outputs the difference information "the performance of each node meets the demand", and then the model analysis module associates the load state of each node to determine the shortage, and obtains that each node "has no resource shortage", the model adjustment module determines the adjustment direction of each node "maintain current performance, optimize transmission stability" according to the no shortage condition, the model integration module summarizes the adjustment direction to form the performance target instruction "node A: maintain delay and bandwidth, optimize transmission stability; node B: maintain delay and bandwidth, optimize transmission stability; node C: maintain delay and bandwidth, optimize transmission stability"; the fourth step calculates the resource performance target parameters based on the performance target instruction: set the load influence coefficient corresponding to the medium load as 1.05, calculate the response delay threshold of each node according to the formula "response delay threshold=real-time response delay data×load influence coefficient", obtain the response delay threshold of node A=22.8×1.05=23.94 ms, the response delay threshold of node B=30.2×1.05=31.71 ms, because 31.71 ms exceeds the demand bottom line of 30 ms, the response delay threshold of node B is adjusted to 30 ms; the response delay threshold of node C=21.5×1.05=22.575 ms, calculate the data transmission bandwidth quota of each node according to the formula "data transmission bandwidth quota=real-time data transmission bandwidth data×load influence coefficient", obtain the data transmission bandwidth quota of node A=68.27×1.05≈71.68 MB / s, the data transmission bandwidth quota of node B=61.8×1.05≈64.89 MB / s, the data transmission bandwidth quota of node C=74.9×1.05≈78.65 MB / s, calculate the algorithm support quota of each node according to the formula "algorithm support quota=real-time node algorithm data×load influence coefficient", obtain the algorithm support quota of node A=33.6×1.05=35.28, the algorithm support quota of node B=24.5×1.05≈25.73, the algorithm support quota of node C=35.2×1.05≈37.0; the fifth step is to aggregate the response delay threshold, data transmission bandwidth quota and computing power support quota of each node to form a storage resource allocation target value.

[0089] Finally, the first step is based on the third step of the generated storage resource allocation target value, the calculation of each node resource demand list: according to the task bearing proportion, that is, node A bears 40%, B bears 35%, C bears 25%, according to the formula "storage capacity = task data processing scale × bearing proportion", the storage capacity of each node is calculated, the storage capacity of node A = 100GB × 40% = 40GB, the storage capacity of node B = 100GB × 35% = 35GB, the storage capacity of node C = 100GB × 25% = 25GB, combined with bandwidth quota, computing power quota, integrated to form resource demand list: node A "storage capacity 40GB, read-write throughput 71.68MB / s, resource type and quantity: bandwidth 71.68MB / s, computing power 35.28, storage space 40GB"; Similarly, the resource demand list of node B and C is formed; The second step is to collect the real-time available resource data of all nodes, and the global available resource list is formed: node A available bandwidth 25MB / s, available computing power 8, available storage 100GB; Node B available bandwidth 30MB / s, available computing power 10, available storage 80GB; Node C available bandwidth 20MB / s, available computing power 12, available storage 120GB; Global available bandwidth = 25+30+20=75MB / s; Global available computing power = 8+10+12=32; Global available storage = 100+80+120=300GB; The third step is to compare the resource demand list and the global available resource list, and to calculate the supply-demand gap: according to the formula "supply-demand gap = total resource demand - global resource supply total", the global resource supply total of each node is calculated, the global bandwidth demand = 71.68+64.89+78.65=215.22MB / s, the bandwidth gap = 215.22-75=140.22MB / s; Global computing power demand = 35.28+25.73+37.0=98.01, computing power gap = 98.01-32=66.01; Global storage demand = 40+35+25=100GB, storage surplus = 300-100=200GB; The fourth step is to determine the resource allocation order and the actual allocation amount based on the supply-demand gap: the preset resource importance level is "bandwidth resource > computing power resource > storage resource", according to the "demand value large priority" sorting rule, the bandwidth allocation sub-order C→A→B, the computing power allocation sub-order C→A→B, the storage allocation sub-order A→B→C, and the resource allocation order is formed in series, which is "first, according to C→A→B to allocate bandwidth, then, according to C→A→B to allocate computing power, finally, according to A→B→C to allocate storage"; The fifth step is to calculate the actual allocation amount of each node according to the formula "remaining allocatable amount = total supply amount - allocated cumulative amount": when allocating bandwidth, the demand of node C is 78.65MB / s, total supply 75MB / s, first allocation cumulative amount 0, resulting in remaining allocatable amount = 75-0 = 75MB / s, node C demand is greater than the remaining allocatable amount, and the load bandwidth carrying upper limit is 80MB / s, then the actual allocation of node C is 75MB / s, and the cumulative amount is updated to 75, the bandwidth demand of nodes A and B exceeds the remaining allocatable amount, and it is planned to allocate 71.68MB / s and 64.89MB / s from the reserved resource pool respectively, and the computing power is allocated in the same way, resulting in node C allocation 12, A allocation 8, B allocation 10, then the gap is filled from the reserved pool and stored, and according to the demand allocation, each node has no gap; the sixth step is to integrate the resource scheduling strategy and generate real-time control signals: node A scheduling strategy "allocation path: reserved resource pool→A; adjustment range: bandwidth + 71.68MB / s, computing power + 35.28, storage + 40GB; implementation time: T+10 seconds"; similarly, the scheduling strategies of nodes B and C are formed, and the scheduling strategies are converted into instruction parameters, such as node A "allocation path: Pool→A; bandwidth adjustment value: 71.68; computing power adjustment value: 35.28; storage adjustment value: 40; implementation time: T+10", and the real-time control signals are encoded in JSON format, such as {"node":"A","path":"Pool→A","bandwidth":71.68,"computing":35.28,"storage":40,"time":"T+10"}, and the real-time control signals are sent to each node and the reserved resource pool, and the storage resource dynamic allocation between the storage nodes is performed.

[0090] Figure 3 A specific implementation structure diagram of a storage resource dynamic allocation system based on artificial intelligence provided by an embodiment of the present application is shown in Figure 3 The system can include: The acquisition module 31 is configured to acquire real-time response delay data, real-time data transmission bandwidth data and real-time node computing power data of each storage node in a target task stage of a target task.

[0091] The processing module 32 is configured to cooperatively process the real-time response delay data, the real-time data transmission bandwidth data and the real-time node computing power data to obtain a storage resource real-time performance data packet containing real-time load states of each storage node.

[0092] The analysis module 33 is configured to perform performance analysis on the storage resource real-time performance data packet based on an artificial intelligence model corresponding to a task stage to obtain a storage resource allocation target value.

[0093] The generating module 34 is configured to formulate a resource scheduling strategy based on the storage resource allocation target value by using a dynamic allocation optimization algorithm, and generate a real-time control signal corresponding to the resource scheduling strategy, and perform dynamic allocation and scheduling of the storage resources among the storage nodes based on the real-time control signal.

[0094] The storage resource dynamic allocation system based on artificial intelligence provided in the embodiments of the present application is used to implement the storage resource dynamic allocation method based on artificial intelligence as described above, and therefore the specific embodiments of the storage resource dynamic allocation system based on artificial intelligence can refer to the embodiments of the storage resource dynamic allocation method based on artificial intelligence as described above, and the specific embodiments can refer to the descriptions of the respective embodiments, which will not be repeated here.

[0095] As shown in Figure 4 The present application further provides an electronic device, which comprises a memory for storing a computer program and a processor for executing the computer program to implement the steps of the storage resource dynamic allocation method based on artificial intelligence as described above.

[0096] The present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the storage resource dynamic allocation method based on artificial intelligence as described above. In an exemplary embodiment, the computer readable storage medium as described above can include but is not limited to a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0097] The embodiments of the present application further provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the storage resource dynamic allocation method based on artificial intelligence as described above.

[0098] Those skilled in the art can further appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0099] The above describes in detail the method and system for dynamically allocating storage resources based on artificial intelligence provided by the present application. The principles and implementation modes of the present application are described by applying specific examples, and the above description of the examples is only used to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method for dynamically allocating storage resources based on artificial intelligence, characterized in that, include: Obtain real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task; The real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data are processed together to obtain a real-time performance data packet of storage resources that includes the real-time load status of each storage node. Based on the artificial intelligence model corresponding to the task stage, the real-time performance data packet of the storage resources is analyzed to obtain the target value for storage resource allocation. The dynamic allocation optimization algorithm is used to formulate a resource scheduling strategy based on the target value of the storage resource allocation, and generate a real-time control signal corresponding to the resource scheduling strategy. Based on the real-time control signal, the dynamic allocation and scheduling of storage resources among the storage nodes are executed.

2. The method according to claim 1, characterized in that, The step of formulating a resource scheduling strategy based on the target value of storage resource allocation using a dynamic allocation optimization algorithm, and generating a real-time control signal corresponding to the resource scheduling strategy, includes: Based on the storage resource allocation target value, and combined with the hardware specifications and task load characteristics of each storage node, calculate the resource requirement list of each storage node under the current task stage. The resource requirement list includes: storage capacity, read / write throughput, storage resource type and quantity. Collect real-time available resource data from all storage nodes and compile them into a global available resource list. Compare the resource demand list of each storage node with the global available resource list item by item. Based on the comparison results, quantify and calculate the supply and demand difference of various resources. The part where demand exceeds supply is the resource gap, and the part where supply exceeds demand is the resource surplus. Based on the supply and demand difference of the various resources, the resource allocation order of each storage node is determined through a dynamic allocation optimization algorithm. According to the resource allocation order, the resource requirement list of each storage node is adjusted to form a resource scheduling strategy for each storage node. The resource scheduling strategy includes: resource allocation path, adjustment range and implementation time. The resource scheduling strategy is transformed into rules to obtain instruction parameters. The instruction parameters are then encoded according to a preset signal format to generate a real-time control signal corresponding to the resource scheduling strategy.

3. The method according to claim 2, characterized in that, The process of determining the resource allocation order of each storage node based on the supply-demand gap of the various resources using a dynamic allocation optimization algorithm includes: Based on the supply and demand gaps of various resources, the corresponding allocation gaps or surplus information is determined. The allocation gaps or surplus information includes: the type, scale, and scope of impact of the resource gaps or surpluses. The allocation gap or surplus information is classified according to the type of response latency resources, data transmission bandwidth resources, and computing power resources to obtain the gap or surplus degree corresponding to each type of resource. The shortage or surplus of each type of resource is compared with the corresponding preset resource importance level to determine the allocation priority of the corresponding resource in the current task stage. Extract the required value of each type of resource in each storage node and the urgency level of the current task carried by each storage node from the resource requirement list; The required value is associated with the urgency level identifier. Based on the association result, the storage nodes under the same resource are sorted according to the preset sorting rules to obtain the node allocation sub-order corresponding to the resource. According to the allocation priority, all the node allocation sub-orders are concatenated and integrated to obtain the resource allocation order of each storage node.

4. The method according to claim 2, characterized in that, The step of adjusting the resource requirement list of each storage node according to the resource allocation order to form a resource scheduling strategy for each storage node includes: According to the resource allocation order, the resource type with the highest priority is selected to obtain the resource to be processed. According to the sub-allocation order of the nodes corresponding to the resource to be processed, the current processing node is selected in turn. Obtain the current processing node's demand for the resource to be processed, the total supply of the resource to be processed, and the cumulative amount already allocated to other nodes. Calculate the difference between the total supply and the cumulative amount to obtain the remaining allocable amount. If the demand value is less than or equal to the remaining allocable amount, then the demand value is taken as the actual allocation amount of the current processing node; or, if the demand value is greater than the remaining allocable amount, then, in combination with the upper limit of the capacity of the resource to be processed in the real-time load status of the current processing node, the value that does not exceed the remaining allocable amount and does not exceed the upper limit of the capacity is taken as the actual allocation amount of the current processing node. The actual allocation amount is recorded as the final allocation result of the current processing node to the resource to be processed. The final allocation results of each storage node in various types of resources are integrated to form a resource scheduling strategy for each storage node.

5. The method according to claim 1, characterized in that, The process of performing performance analysis on the real-time performance data packets of the storage resources based on the artificial intelligence model corresponding to the task stage to obtain the target value for storage resource allocation includes: Obtain the requirement characteristics corresponding to the current task stage, including the task data processing scale, read and write operation frequency, data persistence requirements, and service quality level; The real-time performance data packets of the storage resources are associated and integrated with the demand characteristics to obtain the data to be analyzed; By using the artificial intelligence model corresponding to the current task stage, the performance of the data to be analyzed is performed to obtain the performance target instructions for each storage node to meet the requirements of the current task stage. Based on the response speed adaptation requirements in the performance target instructions, the real-time response latency data is combined with the real-time load status to obtain the response latency threshold. Based on the data transmission capability adaptation requirements in the performance target instructions, the real-time data transmission bandwidth data is combined with the real-time load status to obtain the data transmission bandwidth quota. Based on the processing capability adaptation requirements in the performance target instructions, the real-time node computing power data is combined with the real-time load status to obtain the computing power support quota. The target value for storage resource allocation for each storage node is obtained by summarizing all the aforementioned response latency thresholds, data transmission bandwidth quotas, and computing power support quotas.

6. The method according to claim 5, characterized in that, The step involves using the artificial intelligence model corresponding to the current task stage to perform performance analysis on the data to be analyzed, and obtaining performance target instructions for each storage node to meet the requirements of the current task stage, including: By using the comparison module in the artificial intelligence model corresponding to the current task stage, the real-time performance data packets of storage resources in the data to be analyzed are compared with the demand characteristics to obtain the difference information between each storage node and the demand characteristics in terms of response speed, data transmission capability, and processing capability. The analysis module in the artificial intelligence model correlates the difference information with the real-time load status of the corresponding storage node to determine the shortage situation that caused the difference. The adjustment module in the artificial intelligence model determines the adjustment direction of each storage node in terms of response speed, data transmission capacity, and processing capacity in response to the shortage situation. The integration module in the artificial intelligence model integrates all the aforementioned adjustment directions to form performance target instructions for each storage node to meet the needs of the current task stage.

7. The method according to claim 1, characterized in that, The step of collaboratively processing the real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node includes: Based on the real-time response delay data, the average delay and delay jitter coefficient within a unit time window are calculated to form a delay feature set; The real-time data transmission bandwidth data is statistically analyzed using a sliding window to calculate bandwidth utilization, peak bandwidth ratio, bandwidth fluctuation frequency, and effective transmission rate, thereby generating a set of bandwidth performance indicators. The real-time node computing power data is normalized, and a computing power load assessment value is constructed by combining the node CPU utilization, memory usage, number of input / output operations per second, and task queue length. Using a three-dimensional collaborative analysis model, the latency feature set, the bandwidth performance index set, and the computing power load evaluation value are spatiotemporally correlated and mapped to generate a comprehensive load index for each storage node, and the load status level is determined based on the comprehensive load index. The latency feature set, bandwidth performance index set, computing power load assessment value, comprehensive load index and load status level of each storage node are structurally integrated to form a real-time performance data packet for storage resources.

8. A dynamic storage resource allocation system based on artificial intelligence, characterized in that, include: The acquisition module is used to acquire real-time response latency data, real-time data transmission bandwidth data, and real-time node computing power data of each storage node at the current task stage of the target task. The processing module is used to collaboratively process the real-time response latency data, the real-time data transmission bandwidth data, and the real-time node computing power data to obtain a real-time performance data packet of storage resources containing the real-time load status of each storage node. The analysis module is used to perform performance analysis on the real-time performance data packets of the storage resources based on the artificial intelligence model corresponding to the task stage, so as to obtain the target value for storage resource allocation. The generation module is used to formulate a resource scheduling strategy based on the storage resource allocation target value through a dynamic allocation optimization algorithm, generate a real-time control signal corresponding to the resource scheduling strategy, and execute the dynamic allocation and scheduling of storage resources among storage nodes based on the real-time control signal.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the AI-based dynamic allocation method for storage resources as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the artificial intelligence-based dynamic allocation method for storage resources as described in any one of claims 1 to 7.

Images