Computer data center resource scheduling method and system based on artificial intelligence

Abstract

The invention discloses a computer data center resource scheduling method and system based on artificial intelligence, and relates to the technical field of data center resource scheduling and storage management, and the method comprises the steps: obtaining a time sequence sample, calculating a physical access time interval through the time sequence sample, carrying out the normalization of the physical access time interval, and obtaining a normalized access time interval; constructing a log-likelihood function based on maximum likelihood estimation, calculating an iterative update step length by adopting a Newton-Raphson incremental algorithm in combination with a fractional order log-bound function, and iteratively updating a shape distribution parameter and a scale distribution parameter; inputting the normalized access time interval, the optimal shape distribution parameter and the optimal scale distribution parameter into a risk assessment model, calculating an instantaneous risk rate, converting the instantaneous risk rate into a target access probability accessed in the next unit time, and constructing an inequality constraint trigger by combining the node retention penalty cost of a target data block and the network migration physical cost; and distributing the target data block to the matched target node.

Description

Technical Field

[0001] This invention relates to the technical field of data center resource scheduling and storage management, and in particular to a computer data center resource scheduling method and system based on artificial intelligence. Background Technology

[0002] In existing data center resource scheduling (such as container migration, data tiering, and cross-node memory page migration), a strategy of setting a fixed access count threshold is commonly used to trigger data migration. However, this type of technical solution has three inherent drawbacks: First, the fixed threshold mechanism cannot adapt to dynamically fluctuating workloads and is prone to failure and ineffective migration storms during network congestion or sudden high concurrency. Second, existing probability distribution scheduling models incorrectly follow the conventional assumption that the longer a data is not accessed, the colder it becomes, and completely fail to capture the bathtub curve characteristics of batch processing tasks, causing the system to be unable to pull data back to the hot layer in advance before the peak of batch processing. Finally, both deploying complex deep learning computing frameworks and maintaining independent access counters for massive underlying data blocks will bring huge computing power and memory overhead, making it impossible for existing technologies to achieve efficient real-time scheduling in resource-constrained underlying firmware or kernel space.

[0003] Currently, Chinese invention patent application number CN202411962585.5 discloses a resource scheduling system for a prefabricated data center, comprising: obtaining configuration topology information of multiple prefabricated units through interactive target scenarios; making initial static scheduling decisions based on real-time task requirements to generate an initial resource scheduling strategy set; parsing the configuration topology information to obtain connectivity characteristics of multiple prefabricated units and real-time resource status information; performing iterative decision optimization of the initial resource scheduling strategy set to obtain an optimized scheduling strategy set; selecting the top N optimal optimized scheduling strategies from the optimized scheduling strategy set, outputting a resource scheduling plan set, and performing resource scheduling control of the target prefabricated data center according to the resource scheduling plan set. Existing data center scheduling methods rely on fixed thresholds, which makes them unable to adapt to dynamic loads, unable to capture the bathtub curve access characteristics generated by batch processing tasks, and suffer from the technical problems of huge system overhead caused by maintaining a global access counter and using complex deep learning models. Summary of the Invention

[0004] The technical problem solved by this invention is that existing data center scheduling methods rely on fixed thresholds, which makes them unable to adapt to dynamic loads, unable to capture the bathtub curve access characteristics generated by batch processing tasks, and have huge system overhead due to maintaining a global access counter and using complex deep learning models.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a computer data center resource scheduling method based on artificial intelligence, comprising the following steps: Step S1: Use a sliding window to collect the identifier and corresponding timestamp of the accessed target data block to obtain a time series sample; Step S2: Calculate the physical access time interval from the time series samples and normalize it to obtain the normalized access time interval. Using the normalized access time interval as the observation, construct the log-likelihood function based on the maximum likelihood estimation. Use the Newton-Raphson incremental algorithm combined with the fractional log bound function to calculate the iterative update step size. Iteratively update the shape distribution parameters and scale distribution parameters to obtain the optimal shape distribution parameters and optimal scale distribution parameters. Step S3: Input the normalized access time interval, optimal shape distribution parameter and optimal scale distribution parameter into the risk assessment model, calculate the instantaneous risk rate and convert it into the target access probability in the next unit time, and construct an inequality constraint trigger by combining the node retention penalty cost of the target data block and the physical cost of network migration. Step S4: When the inequality constraint trigger is true, a physical migration task is generated, the cumulative quantile value corresponding to the normalized access time interval is calculated, and the target data block is allocated to the matching target node based on the joint mapping result of the target access probability and the cumulative quantile value.

[0006] Preferably, step S1 includes: Three consecutive short-time sliding windows are initialized on the computing nodes of the computer data center, namely the first sliding window, the second sliding window, and the third sliding window; The system monitors the total input and output read / write count per second of the computing node in real time. When the total input and output read / write count per second exceeds the preset load start threshold, the Bloom filter is activated. The first identifier of the first target data block accessed within the first sliding window is mapped using a hash algorithm. If the mapping does not hit the first Bloom filter, it is determined to be a valid tracking. The first identifier and the current timestamp are appended to the first time series queue, and the mapping result is registered in the first Bloom filter to form the first working set. If a hit occurs, skip the record; The second identity identifier of the second target data block accessed within the second sliding window is mapped using a hash algorithm. If the mapping does not hit the second Bloom filter, it is determined to be a valid tracking. The second identity identifier and the current timestamp are appended to the second time series queue, and the mapping result is registered in the second Bloom filter to form the second working set. If a hit occurs, skip the record; The third identity identifier of the third target data block accessed within the third sliding window is mapped using a hash algorithm. If the mapping does not hit the third Bloom filter, it is determined to be a valid tracking. The third identity identifier and the current timestamp are appended to the third time series queue, and the mapping result is registered in the third Bloom filter to form the third working set. If a hit occurs, skip the record; When the sliding window switches and the previous round of time series samples has been extracted, the Bloom filter and the corresponding time series queue entering the new round of sampling are cleared. The first working set, the second working set, and the third working set are merged in time stamp order to form a time series sample. The time series sample includes: the start and end timestamps of each sliding window, and a sequence of tuples including identity identifiers and access timestamps sorted by time. The identifier is the unique address of the target data block in the underlying operating system and storage hardware.

[0007] Preferably, step S2 includes the following sub-steps: Step S21: Extract the most recent access timestamp of the target data block in the time series sample. If there is no access record of the target data block in the time series sample, extract the most recent access timestamp metadata maintained by the node side for the target data block, calculate the difference between the current sampling time and the most recent access timestamp, and obtain the physical access time interval. Step S22: Obtain the factory-predicted maximum lifespan of the computer data center system, and linearly normalize the physical access time interval based on the factory-predicted maximum lifespan to obtain the normalized access time interval. The mathematical expression for linear normalization is: ; ; in, Here, x represents the normalized mapping value, and x is the normalized access time interval constrained within the interval (0,1). Physical access time interval, The maximum lifespan predicted at the time of manufacture. Preset a small tolerance value; Step S23: In the background management node of the computer data center, the normalized access time interval is used as the observation data to initialize the mathematical bathtub distribution model; Step S24: Construct the log-likelihood function based on the observation dataset Calculate the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter, respectively, and generate the gradient vector; The Newton-Raphson incremental algorithm, combined with a fractional logarithmic bound function, is used to calculate the iterative update step size based on the gradient vector. The shape distribution parameters and scale distribution parameters are mathematically updated iteratively. After each iteration, the absolute value of the difference between the current iteration value and the previous iteration value is calculated to obtain the iteration drift. The iteration drift is then compared with a preset convergence threshold. If the absolute value of the difference is less than the preset convergence threshold, or if the number of iterations reaches the preset maximum number of iterations threshold, the operation is considered to have converged. When the computation converges, the mathematical iteration update terminates and the optimal shape distribution parameters and optimal scale distribution parameters for the current data access workload are output. If the operation does not converge, continue with the mathematical iteration update.

[0008] Preferably, in step S23, initializing the mathematical bathtub distribution model specifically includes: The set of normalized access time intervals of n target data blocks acquired within a preset sampling period is used as the observation dataset; The probability density function of the mathematical bathtub distribution model is initialized, and the initial iterative values ​​of the shape distribution parameters and scale distribution parameters are preset. The mathematical expression of the probability density function is: ; in, Let be the probability density function of the mathematical bathtub distribution model. For shape distribution parameters, For scale distribution parameters, To normalize the access time interval, It is the inverse hyperbolic tangent function. It is an exponential function with the natural constant e as its base.

[0009] Preferably, step S24 specifically includes: Step S241: Based on the observed dataset, take the natural logarithm of the probability density function of the mathematical bathtub distribution model and sum them to construct the log-likelihood function, the mathematical expression of which is: ; in, Let be the log-likelihood function. The total number of samples in the observation dataset. The normalized access time interval for the i-th target data block in the observation dataset; Step S242: Calculate the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter respectively, and combine the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter to generate a gradient vector; Step S243: Calculate the second partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter respectively, and combine the second partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter to generate the Hessian matrix; Step S244: Introduce a fractional logarithmic bound function as a dynamic penalty factor, calculate the iterative update step size, multiply the inverse of the Hessian matrix, the gradient vector, and the fractional logarithmic bound function to calculate and obtain the k-th iterative update step size; Step S245: Using the step size updated in the k-th iteration, perform the (k+1)-th mathematical iteration update on the shape distribution parameters and scale distribution parameters. The mathematical expression for the iterative update is: ; in, To update the step size vector for the k-th iteration, To update the shape distribution parameters of the output for the k-th iteration, To update the scale distribution parameters output in the k-th iteration, To update the shape distribution parameters of the output for the (k+1)th iteration, Update the scale distribution parameters output for the (k+1)th iteration; After each iteration update, the sum of the absolute values ​​of the differences between the current iteration value and the previous iteration value is calculated to obtain the iteration drift.

[0010] Preferably, step S244 specifically includes: The calculation process of the fractional-order logarithmic bound function includes: The unconstrained original step size extracted from the product of the inverse of the Hessian matrix and the gradient vector is mathematically expressed as follows: ; in, This represents the original update distance for the shape distribution parameters in the current k-th iteration. This represents the original update distance for the scale distribution parameters in the current k-th iteration, i.e. and This is the unconstrained initial step size; The fractional-order boundary safety factors for shape distribution parameters and scale distribution parameters are calculated independently, and their mathematical expressions are as follows: ; in, The fractional-order boundary safety factor for the shape distribution parameter. The fractional boundary safety factor is the scale distribution parameter. The pre-defined boundary approximation backoff constant; The minimum value between the fractional-order boundary safety factor of the shape distribution parameter and the fractional-order boundary safety factor of the scale distribution parameter is taken as the final output value of the fractional-order logarithmic bound function, and its mathematical expression is: ; in, It is a fractional-order logarithmic bound function; The mathematical expression for calculating the update step size for the k-th iteration is: ; in, It is the inverse of the Hessian matrix. For Hessian matrix, Let k be the gradient vector, and k be the index of the current iteration number.

[0011] Preferably, step S3 specifically includes: The pre-defined risk assessment model is a mathematical bathtub distribution risk function, whose mathematical expression is: ; in, Instantaneous risk rate For optimal shape distribution parameters, The optimal scale distribution parameters; The mathematical expression for converting the instantaneous risk rate into the probability of accessing the target within the next unit of time is: ; ; in, This represents the probability of the target being accessed in the next unit of time. This represents the normalized time increment per unit of time. The preset unit time length; Construct an inequality constraint trigger. The decision logic for the inequality constraint trigger is as follows: ; in, The cost of node delays The physical cost of network migration; The cost of node stagnation penalty is the sum of the additional physical time delays required when the next cross-node read event or error correction code verification event occurs if the target data block has not been migrated. The physical cost of network migration is the sum of the transmission bandwidth time and the local flash memory write time required to immediately migrate the target data block to the nearest available node.

[0012] Preferably, step S4 includes: When the inequality constraint trigger is set, the target data block is determined to enter the state that must be migrated and a physical migration task is generated. The current bandwidth utilization rate of the network port inside the data center is detected. When the current bandwidth utilization rate is less than the preset idle bandwidth threshold, the physical migration task is immediately executed on the target data block. When the current bandwidth utilization rate is greater than or equal to the preset idle bandwidth threshold, the physical migration task will be placed in the execution queue and its execution will be suspended until the idle bandwidth threshold is met. If the inequality constraint trigger does not hold, the target data block is retained in the current node; Once the physical migration task is initiated, a cumulative probability mapping algorithm based on the bathtub distribution is used to continuously map the normalized access time interval corresponding to the target data block to the standard probability space. The cumulative quantile value of the target data block is then calculated. The calculation of the cumulative quantile value is derived based on the cumulative distribution function of the mathematical bathtub distribution model, and its mathematical expression is as follows: ; in, The cumulative quantile value of the target data block; Send a status polling command to all available candidate nodes in the computer data center to read the current wear index and current read / write queue depth of all available candidate nodes respectively. The current wear index is the number of times the underlying flash memory medium of the available candidate node has been erased and written, and the current read / write queue depth is the total number of input / output requests that are currently being queued and waiting to be processed by the available candidate node. The target access probability is compared with the preset extreme risk threshold, and the cumulative quantile value is compared with the preset very early life cycle threshold and very late life cycle threshold respectively, and a two-dimensional joint node placement strategy is executed.

[0013] Preferably, the two-dimensional joint node placement strategy specifically includes: If the target access probability is greater than the preset extreme risk threshold and the cumulative quantile value is less than the preset very early life cycle threshold, the target data block is determined to be in the high-frequency writing period at the beginning of its life cycle. Available candidate nodes with current wear indicators less than the preset very low wear safety threshold and current read / write queue depth of zero are extracted as first-level target nodes. The target data block is then forcibly allocated and migrated to the first-level target node. If the target access probability is greater than the preset extreme risk threshold and the cumulative quantile value is greater than the preset late life cycle threshold, the target data block is determined to be in the second burst period before the batch task arrives. The available candidate nodes with the highest read bandwidth index and within the preset wear tolerance range are selected as the second-level target nodes, and the target data block is allocated and migrated to the second-level target nodes. If the above-mentioned criteria for determining the first-level or second-level target node are not met, the target data block is determined to be non-extreme risk data. Nodes within the normal wear range are selected from the candidate nodes as third-level target nodes, and the target data block is allocated and migrated to the third-level target node.

[0014] An AI-based computer data center resource scheduling system is implemented to execute an AI-based computer data center resource scheduling method. The system includes a sampling module, a parameter update module, a risk assessment module, and a resource scheduling module. The sampling module is used to collect the identifier and corresponding timestamp of the accessed target data block using a sliding window to obtain time series samples; The parameter update module is used to calculate the physical access time interval from the time series samples and normalize it to obtain the normalized access time interval. Using the normalized access time interval as the observation, a log-likelihood function is constructed based on the maximum likelihood estimation. The Newton-Raphson incremental algorithm is combined with the fractional log bound function to calculate the iterative update step size. The shape distribution parameters and scale distribution parameters are iteratively updated to obtain the optimal shape distribution parameters and the optimal scale distribution parameters. The risk assessment module is used to input the normalized access time interval, optimal shape distribution parameters and optimal scale distribution parameters into the risk assessment model, calculate the instantaneous risk rate and convert it into the target access probability in the next unit time, and construct inequality constraint triggers by combining the node retention penalty cost of the target data block with the physical cost of network migration. The resource scheduling module generates physical migration tasks when the inequality constraint trigger is true, calculates the cumulative quantile value corresponding to the normalized access time interval, and allocates the target data block to the matching target node based on the joint mapping result of the target access probability and the cumulative quantile value.

[0015] The beneficial effects of this invention are as follows: This invention achieves extremely low-overhead sampling through a sliding window, and uses maximum likelihood estimation and fractional logarithmic bound functions to lightweightly and prevent overshooting the bathtub distribution characteristics. It innovatively transforms scheduling decisions into an inequality hedging between target access probability and physical time cost, and combines cumulative quantile values ​​to achieve two-dimensional precise node placement based on lifecycle and wear level. This reduces system tail latency while maximizing the overall lifespan of the cluster storage medium. Furthermore, this invention is the only effective scheduling method for addressing the "last gasp" phenomenon in batch processing tasks. It can automatically calculate the risk rate of a rapid spike when approaching the limit before the batch processing task arrives, and pull the data back to the hot layer in advance. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the basic process of a computer data center resource scheduling method based on artificial intelligence, provided as an embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Reference Figure 1 As an embodiment of the present invention, a computer data center resource scheduling method based on artificial intelligence is provided, comprising the following steps: Step S1: Use a sliding window to collect the identifier and corresponding timestamp of the accessed target data block to obtain a time series sample; Step S2: Calculate the physical access time interval from the time series samples and normalize it to obtain the normalized access time interval. Using the normalized access time interval as the observation, construct the log-likelihood function based on the maximum likelihood estimation. Use the Newton-Raphson incremental algorithm combined with the fractional log bound function to calculate the iterative update step size. Iteratively update the shape distribution parameters and scale distribution parameters to obtain the optimal shape distribution parameters and optimal scale distribution parameters. Step S3: Input the normalized access time interval, optimal shape distribution parameter and optimal scale distribution parameter into the risk assessment model, calculate the instantaneous risk rate and convert it into the target access probability in the next unit time, and construct an inequality constraint trigger by combining the node retention penalty cost of the target data block and the physical cost of network migration. Step S4: When the inequality constraint trigger is true, a physical migration task is generated, the cumulative quantile value corresponding to the normalized access time interval is calculated, and the target data block is allocated to the matching target node based on the joint mapping result of the target access probability and the cumulative quantile value.

[0019] This method achieves data state awareness with extremely low memory overhead through sliding windows and Bloom filters. By combining maximum likelihood estimation with fractional logarithmic bound functions, it accurately and prevents data access from exceeding limits without deploying a large deep learning framework. It innovatively transforms scheduling decisions into an inequality hedging between target access probability and physical time cost, achieving adaptive optimal scheduling that automatically suppresses invalid migrations when the network is congested and proactively moves data forward when the network is idle. At the same time, by combining a two-dimensional joint mapping of cumulative quantile values ​​and access probabilities, it accurately places data at different lifecycle stages into physical nodes that match wear and queue depth, significantly reducing system tail latency while maximizing the overall lifespan of the data center cluster storage media.

[0020] In this embodiment, the method of this application is applicable to scenarios such as storage tiered scheduling, distributed cache eviction, and virtual machine memory page cross-node migration in data centers.

[0021] The bathtub curve refers to a specific probability distribution pattern, characterized by extremely high values ​​at the beginning of the lifecycle, followed by a rapid decline and a stable minimum value in the middle stage, before a sharp surge again at the end of the lifecycle. In this invention, it is specifically used to describe the data access patterns of batch processing tasks in data centers.

[0022] A Bloom filter is a highly space-efficient probabilistic data structure that uses multiple hash functions to map elements into a bit array for quickly checking whether an element is in a set. Its characteristics include minimal memory usage and constant query time.

[0023] In reliability engineering and survival analysis, the risk assessment model is defined as the instantaneous failure rate or conditional probability of a target event occurring within the next small time interval, given a current survival state. In this invention, it refers to the probability that the target data block will be accessed in the next unit of time.

[0024] Batch processing tasks are large-scale data processing tasks that run automatically in the background on a regular basis within the data center without human intervention. These tasks are characterized by long execution intervals and huge amounts of data read and written in a single operation.

[0025] Maximum likelihood estimation is a statistical parameter estimation method that determines the specific parameters of a probability distribution model by finding parameter values ​​that maximize the probability of observed data.

[0026] Step S1 includes: Three consecutive short-time sliding windows are initialized on the computing nodes of the computer data center, namely the first sliding window, the second sliding window, and the third sliding window; The system monitors the total input and output read / write count per second of the computing node in real time. When the total input and output read / write count per second exceeds the preset load start threshold, the Bloom filter is activated. The first identifier of the first target data block accessed within the first sliding window is mapped using a hash algorithm. If the mapping does not hit the first Bloom filter, it is determined to be a valid tracking. The first identifier and the current timestamp are appended to the first time series queue, and the mapping result is registered in the first Bloom filter to form the first working set. If a hit occurs, skip the record; The second identity identifier of the second target data block accessed within the second sliding window is mapped using a hash algorithm. If the mapping does not hit the second Bloom filter, it is determined to be a valid tracking. The second identity identifier and the current timestamp are appended to the second time series queue, and the mapping result is registered in the second Bloom filter to form the second working set. If a hit occurs, skip the record; The third identity identifier of the third target data block accessed within the third sliding window is mapped using a hash algorithm. If the mapping does not hit the third Bloom filter, it is determined to be a valid tracking. The third identity identifier and the current timestamp are appended to the third time series queue, and the mapping result is registered in the third Bloom filter to form the third working set. If a hit occurs, skip the record; When the sliding window switches and the previous round of time series samples has been extracted, the Bloom filter and the corresponding time series queue entering the new round of sampling are cleared. The first working set, the second working set, and the third working set are merged in time stamp order to form a time series sample. The time series sample includes: the start and end timestamps of each sliding window, and a sequence of tuples including identity identifiers and access timestamps sorted by time. The identifier is the unique address of the target data block in the underlying operating system and storage hardware.

[0027] In this embodiment, three consecutive short-time sliding windows, each with a duration of 100 milliseconds, are established.

[0028] Specifically, when scheduling occurs at the storage layer, such as solid-state drives, the identity identifier corresponds to the logical block address (LBA). When scheduling occurs at the memory level, the identity identifier corresponds to the physical page frame number (PFN). In a distributed file system, the identity identifier corresponds to the object identifier ObjectID; Bloom filters are used to remove duplicates within a window, meaning that only the first access event is recorded for the same identity identifier within the same sliding window. The first working set, the second working set, and the third working set are each composed of a corresponding Bloom filter and a corresponding time series queue. The Bloom filter is used for gating and deduplication, and the time series queue is used to store (identifier, access timestamp) tuples.

[0029] As long as the target data block is accessed within the corresponding window, its identifier is hashed and mapped to the corresponding Bloom filter. This allows for the approximate recording of the time period in which the data appeared with extremely low space overhead (allowing for a very small probability of false positives). False positives from the Bloom filter may cause some first-time access events to be misjudged as hits, thus skipping the recording. The probability of missing records is controlled within a preset upper limit by the Bloom filter parameter configuration, preferably controlled at [preferred upper limit]. to between.

[0030] Optionally, the load trigger threshold is set based on one of IOPS, queue depth, or utilization. When IOPS is used as the trigger, the threshold is 60% to 90% of the device's rated IOPS. When the queue depth is used as the trigger, the threshold is set to a queue depth greater than or equal to 1 to 4. When utilization is triggered, the threshold is set to a device busyness level of 70% to 95%.

[0031] Step S1 leverages the strongest engineering advantage of the segmented scheduling method: on the computing nodes of the data center, it avoids maintaining a massive access counter for each data block or memory page, thus preventing the consumption of huge amounts of memory. Using the first, second, and third working sets, a Bloom filter with extremely low space complexity is used to obtain time-series samples of data access, completing data state awareness and data preparation with minimal overhead for the subsequent mathematical calculations of the risk model.

[0032] This step uses a Bloom filter as the initial screening gate, recording and timestamping only the first access event that fails to reach the target. This mechanism achieves accurate perception of high-frequency data status with extremely low space complexity and minimal memory overhead, solving the industry problem of traditional scheduling methods maintaining a global counter for each data block, thus consuming host machine memory.

[0033] Step S2 includes the following sub-steps: Step S21: Extract the most recent access timestamp of the target data block in the time series sample. If there is no access record of the target data block in the time series sample, extract the most recent access timestamp metadata maintained by the node side for the target data block, calculate the difference between the current sampling time and the most recent access timestamp, and obtain the physical access time interval. Step S22: Obtain the factory-predicted maximum lifespan of the computer data center system, and linearly normalize the physical access time interval based on the factory-predicted maximum lifespan to obtain the normalized access time interval. The mathematical expression for linear normalization is: ; ; in, Here, x represents the normalized mapping value, and x is the normalized access time interval constrained within the interval (0,1). Physical access time interval, The maximum lifespan predicted at the time of manufacture. Preset a small tolerance value; In this embodiment, the unit of the factory-predicted maximum lifespan duration is... Maintain consistency, preset small tolerance value Values ; Step S23: In the background management node of the computer data center, the normalized access time interval is used as the observation data to initialize the mathematical bathtub distribution model; Step S24: Construct the log-likelihood function based on the observation dataset Calculate the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter, respectively, and generate the gradient vector; The Newton-Raphson incremental algorithm, combined with a fractional logarithmic bound function, is used to calculate the iterative update step size based on the gradient vector. The shape distribution parameters and scale distribution parameters are mathematically updated iteratively. After each iteration, the absolute value of the difference between the current iteration value and the previous iteration value is calculated to obtain the iteration drift. The iteration drift is then compared with a preset convergence threshold. If the absolute value of the difference is less than the preset convergence threshold, or if the number of iterations reaches the preset maximum number of iterations threshold, the operation is considered to have converged. When the computation converges, the mathematical iteration update terminates and the optimal shape distribution parameters and optimal scale distribution parameters for the current data access workload are output. If the operation does not converge, continue with the mathematical iteration update.

[0034] In this embodiment, the preset convergence threshold is set based on a balance between the precision requirements of the underlying CPU's floating-point unit (FPU) and the scheduling system's time budget for microsecond-level computation latency, and its value range is [range missing]. to When the sum of the absolute values ​​of the parameter differences between two iterations falls into this tiny interval, it is determined that the mathematical distribution has infinitely approximated the actual workload. Continuing the iteration yields marginal benefits for scheduling decisions that approach zero, making it the most economical time to exit the iteration. The maximum number of iterations threshold is 20–50.

[0035] The optimal shape distribution parameters and optimal scale distribution parameters are passed to the risk assessment model in step S3 for invocation.

[0036] Step S2 inherits the strongest advantages of mathematical probability. For time-series samples output by the Bloom filter, it does not require any heavy deep neural network frameworks. Incremental maximum likelihood estimation is performed iteratively using fundamental mathematical formulas, taking only microseconds, making it extremely suitable for online real-time feature extraction and distribution fitting calculations in data centers.

[0037] This step maps the absolute physical time difference to a dimensionless standard open interval, eliminating the dimensional differences between data from different lifecycles. At the same time, by setting a strict convergence threshold for iterative drift and a maximum number of iterations, the microsecond-level time overhead of online computation is strictly limited while ensuring the accuracy of parameter fitting, making it extremely suitable for real-time scheduling of underlying firmware or kernel.

[0038] In step S23, initializing the mathematical bathtub distribution model specifically includes: The set of normalized access time intervals of n target data blocks acquired within a preset sampling period is used as the observation dataset; The probability density function of the mathematical bathtub distribution model is initialized, and the initial iterative values ​​of the shape distribution parameters and scale distribution parameters are preset. The mathematical expression of the probability density function is: ; in, Let be the probability density function of the mathematical bathtub distribution model. For shape distribution parameters, For scale distribution parameters, To normalize the access time interval, It is the inverse hyperbolic tangent function. It is an exponential function with the natural constant e as its base.

[0039] In this embodiment, n is determined based on the law of large numbers and the cache size allocated to the Bloom filter and time series queue. Too small a sample size will lead to overfitting in the maximum likelihood estimation, while too large a sample size will cause online computation timeouts. The value ranges from 1,000 to 50,000 access time interval samples.

[0040] This step introduces a mathematical bathtub distribution probability density function containing an inverse hyperbolic tangent function, breaking the limitations of traditional normal or exponential distributions in characterizing batch processing tasks in data centers. It can perfectly fit the real physical access patterns of the data lifecycle, which are high at both ends and low in the middle, at the mathematical level, laying a mathematical foundation for subsequent accurate risk prediction.

[0041] Step S24 specifically includes: Step S241: Based on the observed dataset, take the natural logarithm of the probability density function of the mathematical bathtub distribution model and sum them to construct the log-likelihood function, the mathematical expression of which is: ; in, Let be the log-likelihood function. The total number of samples in the observation dataset. The normalized access time interval for the i-th target data block in the observation dataset; Step S242: Calculate the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter respectively, and combine the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter to generate a gradient vector; Step S243: Calculate the second partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter respectively, and combine the second partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter to generate the Hessian matrix; Step S244: Introduce a fractional logarithmic bound function as a dynamic penalty factor, calculate the iterative update step size, multiply the inverse of the Hessian matrix, the gradient vector, and the fractional logarithmic bound function to calculate and obtain the k-th iterative update step size; Step S245: Using the step size updated in the k-th iteration, perform the (k+1)-th mathematical iteration update on the shape distribution parameters and scale distribution parameters. The mathematical expression for the iterative update is: ; in, To update the step size vector for the k-th iteration, To update the shape distribution parameters of the output for the k-th iteration, To update the scale distribution parameters output in the k-th iteration, To update the shape distribution parameters of the output for the (k+1)th iteration, Update the scale distribution parameters output for the (k+1)th iteration; After each iteration update, the sum of the absolute values ​​of the differences between the current iteration value and the previous iteration value is calculated to obtain the iteration drift.

[0042] The mathematical expression for the gradient vector is: ; in, The gradient vector, For the first-order partial derivatives of the shape distribution parameters, Let be the first-order partial derivative of the scale distribution parameter, and let the gradient vector represent the search direction for parameter updates; The mathematical expression for the Hessian matrix is: ; in, Let be the Hessian matrix, representing the local curvature of the parameter space. , , and These are the second-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter, respectively.

[0043] The step size vector is updated in the k-th iteration. Directional step size and Directional step size; The mathematical expression for obtaining the iterative drift is: ; in, This represents the iterative drift amount.

[0044] This step transforms the abstract maximum likelihood estimation into matrix operation logic that can be efficiently executed at the computer's low level. By constructing the log-likelihood function and solving for the first-order gradient vector and the second-order Hessian matrix, the convergence speed of parameter optimization is greatly improved by utilizing second-order curvature information, thus meeting the stringent requirements of data centers for extremely low latency scheduling algorithms.

[0045] Step S244 specifically includes: The calculation process of the fractional-order logarithmic bound function includes: The unconstrained original step size extracted from the product of the inverse of the Hessian matrix and the gradient vector is mathematically expressed as follows: ; in, This represents the original update distance for the shape distribution parameters in the current k-th iteration. This represents the original update distance for the scale distribution parameters in the current k-th iteration, i.e. and This is the unconstrained initial step size; The fractional-order boundary safety factors for shape distribution parameters and scale distribution parameters are calculated independently, and their mathematical expressions are as follows: ; in, The fractional-order boundary safety factor for the shape distribution parameter. The fractional boundary safety factor is the scale distribution parameter. The pre-defined boundary approximation backoff constant; The minimum value between the fractional-order boundary safety factor of the shape distribution parameter and the fractional-order boundary safety factor of the scale distribution parameter is taken as the final output value of the fractional-order logarithmic bound function, and its mathematical expression is: ; in, It is a fractional-order logarithmic bound function; In this embodiment, The value range is (0,1), with a preferred value of 0.5; When the original iteration step size attempts to update the parameters to negative or zero... It automatically outputs a dynamic scaling factor that is strictly less than 1, forcing the update step size to be truncated within a safe boundary, ensuring that the parameter space always remains strictly positive definite.

[0046] The mathematical expression for calculating the update step size for the k-th iteration is: ; in, It is the inverse of the Hessian matrix. For Hessian matrix, Let k be the gradient vector, and k be the index of the current iteration number.

[0047] In this embodiment, the fractional logarithmic bound function is a dynamic step size scaling factor constructed based on the principle of interior point method. Its function is to establish a boundary penalty mechanism to prevent the shape distribution parameter or scale distribution parameter from illegally exceeding the bounds by less than or equal to zero during the iteration process of the Newton-Raphson incremental algorithm, thereby avoiding the underlying scheduling system from triggering non-numerical operation crashes.

[0048] In step S24, the optimal shape distribution parameters and scale distribution parameters are found using the Newton-Raphson incremental algorithm, and the shape distribution parameters of the mathematical bathtub distribution model are determined. and scale distribution parameters The value must always be greater than 0 because negative time or negative probability does not exist physically. However, the Newton-Raphson incremental algorithm is random and blind in calculating the gradient and Hessian matrix. It might, in a certain iteration, directly calculate a huge negative step size, changing the shape distribution parameter from 0.5 to something else entirely. It was directly reduced to -1.2, once the shape distribution parameter... When the number becomes negative, the next step in the program is to calculate ln( When the value is not equal to 0, it will directly trigger a non-numerical error, causing the kernel of the entire data center scheduling system to crash. The fractional-order logarithmic bound function is designed to prevent such a crash. The fractional order means that a fraction (i.e., scaling factor) between (0,1] is dynamically calculated. The logarithm means that the penalty value of the logarithmic function increases exponentially as the parameter gets closer to 0. The bound means that a virtual limit is established where the parameter is equal to 0. When the step size calculated by the Newton-Raphson incremental algorithm is safe, the fractional logarithmic bound function outputs 1, that is, no intervention is needed, and it proceeds at full speed; When the step size calculated by the Newton-Raphson incremental algorithm crosses 0, causing the scheduling system to crash, the fractional logarithmic bound function urgently outputs a very small fraction (such as 0.1) to forcibly shorten the step size, making the parameter infinitely close to 0 but never equal to or less than 0.

[0049] This step innovatively introduces a fractional-order logarithmic bound function as an intelligent braking system at the algorithm's underlying level. When the Newton iteration step size is too large, this function dynamically outputs a scaling factor to forcibly truncate the step size, fundamentally eliminating the non-numerical system crash problem caused by shape or scale distribution parameters going out of bounds, and greatly improving the robustness of the scheduling system under extreme workloads and its engineering feasibility.

[0050] Step S3 specifically includes: The pre-defined risk assessment model is a mathematical bathtub distribution risk function, whose mathematical expression is: ; in, Instantaneous risk rate For optimal shape distribution parameters, The optimal scale distribution parameters; The mathematical bathtub distribution risk function is used to output the instantaneous risk rate of a target data block given the current unaccessed duration. The mathematical expression for converting the instantaneous risk rate into the probability of accessing the target within the next unit of time is: ; ; in, This represents the probability of the target being accessed in the next unit of time. This represents the normalized time increment per unit of time. The preset unit time length; Construct an inequality constraint trigger. The decision logic for the inequality constraint trigger is as follows: ; in, The cost of node delays The physical cost of network migration; The cost of node stagnation penalty is the sum of the additional physical time delays required when the next cross-node read event or error correction code verification event occurs if the target data block has not been migrated. The physical cost of network migration is the sum of the transmission bandwidth time and the local flash memory write time required to immediately migrate the target data block to the nearest available node.

[0051] In this embodiment, the scheduler no longer arbitrarily judges whether the data is the hottest or the second hottest. Instead, it calculates the probability of the target data block being accessed in the next unit of time, which would cause a tail delay error, if the target data block is not moved. It multiplies this probability by the penalty cost of the error and then calculates whether the overall expected loss is greater than the physical network and input / output cost of moving it away immediately. The preset unit time length is used to convert the instantaneous risk rate into a conditional probability within a specific time window. Its value depends on the granularity of the system's scheduling cycle and ranges from 1 millisecond to 1 second (e.g., set to 100ms). Those skilled in the art can set this based on the clock interrupt frequency of the data center CPU.

[0052] The physical cost of network migration is calculated by adding the local flash write latency estimated by the target node based on the end-to-end transmission latency (PingRTT) in the current network topology detected in real time by the control node (e.g., network transmission 2ms + flash write 3ms = 5ms).

[0053] The cost of node congestion penalty is based on the tail delay penalty coefficient defined in the Service Level Agreement (SLA) or the average blocking time caused by cross-node forced reads in historical logs. For example, forced cross-node reads can cause a long tail block of 50ms.

[0054] The left side of the inequality represents the probability of an error occurring multiplied by the time penalty incurred by the error, i.e., the expected time loss. The right side represents the exact time cost of immediately executing the migration. Migration is only executed when the expected loss exceeds the exact cost. This inequality-constrained trigger completely eliminates the curse of threshold tuning. When the computer data center network is congested, the physical cost of network migration increases, making it difficult for the inequality to hold. The scheduling system automatically becomes lazy to reduce meaningless migrations. When the computer data center network is idle, the physical cost of network migration decreases drastically, making the inequality easily hold. The scheduling system automatically moves potentially risky data in advance. Without manual intervention, the scheduling system achieves adaptive optimality through a purely probabilistic decision trigger constrained by physical overhead.

[0055] This invention innovatively captures the phenomenon of data center batch processing tasks experiencing a brief resurgence. For batch processing tasks that are read aggressively every few hours, existing probability distributions incorrectly assume that the longer the data has not been accessed, the less migration is needed. However, the mathematical bathtub distribution risk function, which includes the inverse hyperbolic tangent function, used in this step can accurately calculate that when the normalized access time interval approaches the right limit of the value, the target risk assessment value will experience a second sharp spike. As a result, the inequality constraint trigger is set again just before the batch processing task arrives, and the scheduling system automatically pulls the data back to the hot storage layer in advance.

[0056] In survival analysis theory, any complete probability distribution (i.e., the mathematical bathtub distribution model in this patent) includes three core mathematical functions, which can be derived from each other through calculus. The three core mathematical functions include the probability density function, the survival function, and the risk function. The probability density function describes the global absolute probability that the data is accessed at a certain absolute time point x. The survival function describes the probability that the data survives past time point x and has not been accessed. The risk function, which is the risk assessment model in this application, describes the instantaneous risk that the data is suddenly accessed in the next second. The risk assessment model is the ratio of the probability density function of the mathematical bathtub distribution model to the survival function of the mathematical bathtub distribution model.

[0057] Mathematically, maximum likelihood estimation (MLE) can only construct a log-likelihood function based on the probability density function. Since the risk function cannot be directly used to fit historical data, in step S2, a mathematical bathtub distribution model must be invoked to complete the parameter training and convergence.

[0058] The scheduler is truly concerned with conditional probabilities. For example, what is the probability that a data block, after being idle for 10 hours, will be accessed at the 10th hour and 1st second? Only the risk assessment model can output this instantaneous lethal probability. The shape distribution parameters trained in step S2... and scale distribution parameters By substituting this into the risk assessment model, the risk value can be accurately calculated, and then offset against the physical relocation cost through an inequality.

[0059] In this invention, the risk assessment model is a mathematically derived function of the mathematical bathtub distribution model. The mathematical bathtub distribution model (expressed as a probability density function) is used at the background control node to perform maximum likelihood estimation on historical observation data to extract distribution parameters; the risk assessment model (expressed as a risk function), by receiving the aforementioned distribution parameters, is used at the computing node to output the instantaneous conditional probability of the target data block being accessed in the next small time interval, given the current period of inaccessibility. Together, they constitute a complete mathematical closed loop from historical parameter fitting to real-time probability prediction.

[0060] This step overturns the traditional rigid scheduling that relies on static fixed thresholds (such as accessing hot data N times), transforming decision-making into a microeconomic hedging of expected penalty loss and exact migration cost. This allows the system to automatically become lazy when the network is congested to suppress ineffective migration, and to proactively migrate when the network is idle, achieving purely probabilistic adaptive optimal decision-making with zero manual parameter tuning.

[0061] Step S4 includes: When the inequality constraint trigger is set, the target data block is determined to enter the state that must be migrated and a physical migration task is generated. The current bandwidth utilization rate of the network port inside the data center is detected. When the current bandwidth utilization rate is less than the preset idle bandwidth threshold, the physical migration task is immediately executed on the target data block. When the current bandwidth utilization rate is greater than or equal to the preset idle bandwidth threshold, the physical migration task will be placed in the execution queue and its execution will be suspended until the idle bandwidth threshold is met. If the inequality constraint trigger does not hold, the target data block is retained in the current node; Once the physical migration task is initiated, a cumulative probability mapping algorithm based on the bathtub distribution is used to continuously map the normalized access time interval corresponding to the target data block to the standard probability space. The cumulative quantile value of the target data block is then calculated. The calculation of the cumulative quantile value is derived based on the cumulative distribution function of the mathematical bathtub distribution model, and its mathematical expression is as follows: ; in, The cumulative quantile value of the target data block; Send a status polling command to all available candidate nodes in the computer data center to read the current wear index and current read / write queue depth of all available candidate nodes respectively. The current wear index is the number of times the underlying flash memory medium of the available candidate node has been erased and written, and the current read / write queue depth is the total number of input / output requests that are currently being queued and waiting to be processed by the available candidate node. The target access probability is compared with the preset extreme risk threshold, and the cumulative quantile value is compared with the preset very early life cycle threshold and very late life cycle threshold respectively, and a two-dimensional joint node placement strategy is executed.

[0062] In this embodiment, the preset extreme risk threshold is the probability value within (0,1), which is 0.80 to 0.99; The preset very early lifecycle threshold is obtained by differentiating the cumulative probability distribution slope of the steep slope on the left side of the bathtub curve (i.e., the early lifecycle when data is frequently overwritten when it is first generated), and extracting the cumulative distribution probability value corresponding to the slope change point as the threshold. The range is 0.05 to 0.20. Data falling within this range is usually very hot metadata that has just been created or modified, which consumes flash memory erase and write life very quickly and must be forced to be placed on a very low wear node.

[0063] The preset late-lifecycle threshold is determined by differentiating the slope of the cumulative probability distribution of the steep right side of the bathtub curve (i.e., the stage where access rates suddenly surge before the batch task arrives after a long period of data inactivity). The probability value corresponding to the abrupt change in the right slope is extracted, ranging from 0.85 to 0.95. Data falling within this range exhibits a "last gasp" characteristic, meaning that although the physical access time interval is extremely long, after being amplified by the inverse hyperbolic tangent function, its quantile value is approaching the limit, requiring early relocation to a high-bandwidth node to cope with the upcoming scan surge.

[0064] This step ensures that physical relocation is performed only under the dual conditions of available network bandwidth, avoiding exacerbating network storms; at the same time, by using the cumulative distribution function to continuously map discrete normalized time to the standard probability space, it accurately characterizes the absolute stage of the data's current lifecycle, providing a reliable mathematical benchmark for subsequent refined placement.

[0065] The two-dimensional joint node placement strategy specifically includes: If the target access probability is greater than the preset extreme risk threshold and the cumulative quantile value is less than the preset very early life cycle threshold, the target data block is determined to be in the high-frequency writing period at the beginning of its life cycle. Available candidate nodes with current wear indicators less than the preset very low wear safety threshold and current read / write queue depth of zero are extracted as first-level target nodes. The target data block is then forcibly allocated and migrated to the first-level target node. If the target access probability is greater than the preset extreme risk threshold and the cumulative quantile value is greater than the preset late life cycle threshold, the target data block is determined to be in the second burst period before the batch task arrives. The available candidate nodes with the highest read bandwidth index and within the preset wear tolerance range are selected as the second-level target nodes, and the target data block is allocated and migrated to the second-level target nodes. If the above-mentioned criteria for determining the first-level or second-level target node are not met, the target data block is determined to be non-extreme risk data. Nodes within the normal wear range are selected from the candidate nodes as third-level target nodes, and the target data block is allocated and migrated to the third-level target node.

[0066] The scheduling system sends status polling commands to the underlying storage medium (such as NVMe solid-state drives), reads the SMART health metadata of the medium, and extracts the percentage of the consumed erase / write cycles (P / ECycles) relative to the factory-nominated maximum erase / write cycles. The preset range for the extremely low wear safety threshold is set to 0%~15%. The first-level target node is used to handle high-frequency write data (extremely hot data) in the very early stages of its life cycle. This type of data will rapidly consume the flash memory's lifespan, so it must be placed on almost brand-new nodes.

[0067] The preset wear tolerance range is set to 60%~90%. The second-level target nodes are used to handle data in the very late stage of its lifecycle (just before batch processing). This type of data is characterized by explosive pure reads and very few writes. Placing them on older nodes with high wear but idle read bandwidth can meet the read surges of batch processing tasks without consuming the remaining write life of older nodes, perfectly achieving cluster-level global wear leveling.

[0068] Correspondingly, the normal wear range is the middle zone between the two mentioned above, namely 15% to 60%; The node placement strategy involves the highest read bandwidth metric, which is derived from the device bandwidth utilization or NVMe underlying statistical throughput periodically reported by each available candidate node. The moving average is calculated using the exponential smoothing algorithm (EMA) to eliminate instantaneous network jitter interference.

[0069] Quantile regression was used to achieve quantile target placement based on wear and load perception. When the trigger determines that migration is necessary, it perfectly inherits the idea of ​​extending flash memory lifespan, but it has evolved from the original four rigid heat classification levels to a continuous quantile matching process based on the target risk assessment value.

[0070] Step S4 introduces the cumulative distribution function as a mathematical mapping tool for quantile regression, achieving quantile target placement based on wear and load perception. Once the inequality trigger determines that migration is necessary, this method abandons the existing approach of pre-setting a fixed number of accesses to divide the data into four rigid discrete levels: cold, warm, hot, and extremely hot. Instead, it precisely evolves this into a continuous quantile (pure decimals between 0 and 1) matching process based on the target risk assessment value. This mechanism ensures that the most frequently accessed and most likely to cause system tail latency dangerous data is always placed on high-quality physical nodes with the lowest wear and no queuing congestion, maximizing the lifespan of the overall data center storage cluster from the physical media level.

[0071] This step enables refined target placement based on wear and load awareness. Through a two-dimensional joint determination using probability and quantile values, very early high-frequency write data is forcibly isolated on nodes with extremely low wear, while very late batch read data is placed on nodes within the wear tolerance range. This strategy perfectly matches the read / write characteristics of data at different lifecycle stages, achieving cluster-level global wear leveling at the physical media level.

[0072] An AI-based computer data center resource scheduling system is implemented to execute an AI-based computer data center resource scheduling method. The system includes a sampling module, a parameter update module, a risk assessment module, and a resource scheduling module. The sampling module is used to collect the identifier and corresponding timestamp of the accessed target data block using a sliding window to obtain time series samples; The parameter update module is used to calculate the physical access time interval from the time series samples and normalize it to obtain the normalized access time interval. Using the normalized access time interval as the observation, a log-likelihood function is constructed based on the maximum likelihood estimation. The Newton-Raphson incremental algorithm is combined with the fractional log bound function to calculate the iterative update step size. The shape distribution parameters and scale distribution parameters are iteratively updated to obtain the optimal shape distribution parameters and the optimal scale distribution parameters. The risk assessment module is used to input the normalized access time interval, optimal shape distribution parameters and optimal scale distribution parameters into the risk assessment model, calculate the instantaneous risk rate and convert it into the target access probability in the next unit time, and construct inequality constraint triggers by combining the node retention penalty cost of the target data block with the physical cost of network migration. The resource scheduling module generates physical migration tasks when the inequality constraint trigger is true, calculates the cumulative quantile value corresponding to the normalized access time interval, and allocates the target data block to the matching target node based on the joint mapping result of the target access probability and the cumulative quantile value.

[0073] This invention abandons the complexities of deep learning frameworks and creatively inputs low-overhead working set sampling data based on Bloom filters directly into a risk assessment model that perfectly fits the data access bathtub curve, constructing a purely probabilistic decision trigger based on physical overhead constraints. This method eliminates the disaster of manually setting rigid soft and hard thresholds. The scheduling system no longer pre-classifies data into hot and cold categories, but dynamically calculates the fatal probability of data being accessed in the next second, leading to tail latency errors. By comparing the dynamic risk probability with the real-time physical migration cost, adaptive optimal scheduling is achieved, automatically reducing migration during network congestion and automatically moving potentially risky data in advance when the network is idle. Furthermore, this invention is an effective scheduling method for the "last gasp" phenomenon in batch processing tasks. It can automatically calculate the rapidly escalating risk rate when approaching the limit before the batch processing task arrives, pulling the data back to the hot layer in advance. The entire method utilizes Bloom filters with kilobyte-level memory usage when extracting features and directly substitutes mathematical formulas with microsecond-level CPU cycles when calculating probabilities, resulting in extremely low space and time complexity. It fully supports direct integration and deployment in the underlying firmware or kernel space.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A computer data center resource scheduling method based on artificial intelligence, characterized in that, Includes the following steps: Step S1: Use a sliding window to collect the identifier and corresponding timestamp of the accessed target data block to obtain a time series sample; Step S2: Calculate the physical access time interval from the time series samples and normalize it to obtain the normalized access time interval. Using the normalized access time interval as the observation, construct the log-likelihood function based on the maximum likelihood estimation. Use the Newton-Raphson incremental algorithm combined with the fractional log bound function to calculate the iterative update step size. Iteratively update the shape distribution parameters and scale distribution parameters to obtain the optimal shape distribution parameters and optimal scale distribution parameters. Step S3: Input the normalized access time interval, optimal shape distribution parameter and optimal scale distribution parameter into the risk assessment model, calculate the instantaneous risk rate and convert it into the target access probability in the next unit time, and construct an inequality constraint trigger by combining the node retention penalty cost of the target data block and the physical cost of network migration. Step S4: When the inequality constraint trigger is true, a physical migration task is generated, the cumulative quantile value corresponding to the normalized access time interval is calculated, and the target data block is allocated to the matching target node based on the joint mapping result of the target access probability and the cumulative quantile value.

2. The computer data center resource scheduling method based on artificial intelligence as described in claim 1, characterized in that, Step S1 includes: Three consecutive short-time sliding windows are initialized on the computing nodes of the computer data center, namely the first sliding window, the second sliding window, and the third sliding window; The system monitors the total input and output read / write count per second of the computing node in real time. When the total input and output read / write count per second exceeds the preset load start threshold, the Bloom filter is activated. The first identifier of the first target data block accessed within the first sliding window is mapped using a hash algorithm. If the mapping does not hit the first Bloom filter, it is determined to be a valid tracking. The first identifier and the current timestamp are appended to the first time series queue, and the mapping result is registered in the first Bloom filter to form the first working set. If a hit occurs, skip the record; The second identity identifier of the second target data block accessed within the second sliding window is mapped using a hash algorithm. If the mapping does not hit the second Bloom filter, it is determined to be a valid tracking. The second identity identifier and the current timestamp are appended to the second time series queue, and the mapping result is registered in the second Bloom filter to form the second working set. If a hit occurs, skip the record; The third identity identifier of the third target data block accessed within the third sliding window is mapped using a hash algorithm. If the mapping does not hit the third Bloom filter, it is determined to be a valid tracking. The third identity identifier and the current timestamp are appended to the third time series queue, and the mapping result is registered in the third Bloom filter to form the third working set. If a hit occurs, skip the record; When the sliding window switches and the previous round of time series samples has been extracted, the Bloom filter and the corresponding time series queue entering the new round of sampling are cleared. The first working set, the second working set, and the third working set are merged in time stamp order to form a time series sample. The time series sample includes: the start and end timestamps of each sliding window, and a sequence of tuples including identity identifiers and access timestamps sorted by time. The identifier is the unique address of the target data block in the underlying operating system and storage hardware.

3. The computer data center resource scheduling method based on artificial intelligence as described in claim 2, characterized in that, Step S2 includes the following sub-steps: Step S21: Extract the most recent access timestamp of the target data block in the time series sample. If there is no access record of the target data block in the time series sample, extract the most recent access timestamp metadata maintained by the node side for the target data block, calculate the difference between the current sampling time and the most recent access timestamp, and obtain the physical access time interval. Step S22: Obtain the factory-predicted maximum lifespan of the computer data center system, and linearly normalize the physical access time interval based on the factory-predicted maximum lifespan to obtain the normalized access time interval. The mathematical expression for linear normalization is: ； ； in, Here, x represents the normalized mapping value, and x is the normalized access time interval constrained within the interval (0,1). Physical access time interval, The maximum lifespan predicted at the time of manufacture. Preset a small tolerance value; Step S23: In the background management node of the computer data center, the normalized access time interval is used as the observation data to initialize the mathematical bathtub distribution model; Step S24: Construct the log-likelihood function based on the observation dataset Calculate the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter, respectively, and generate the gradient vector; The Newton-Raphson incremental algorithm, combined with a fractional logarithmic bound function, is used to calculate the iterative update step size based on the gradient vector. The shape distribution parameters and scale distribution parameters are mathematically updated iteratively. After each iteration, the absolute value of the difference between the current iteration value and the previous iteration value is calculated to obtain the iteration drift. The iteration drift is then compared with a preset convergence threshold. If the absolute value of the difference is less than the preset convergence threshold, or if the number of iterations reaches the preset maximum number of iterations threshold, the operation is considered to have converged. When the computation converges, the mathematical iteration update terminates and the optimal shape distribution parameters and optimal scale distribution parameters for the current data access workload are output. If the operation does not converge, continue with the mathematical iteration update.

4. The computer data center resource scheduling method based on artificial intelligence as described in claim 3, characterized in that, In step S23, initializing the mathematical bathtub distribution model specifically includes: The set of normalized access time intervals of n target data blocks acquired within a preset sampling period is used as the observation dataset; The probability density function of the mathematical bathtub distribution model is initialized, and the initial iterative values ​​of the shape distribution parameters and scale distribution parameters are preset. The mathematical expression of the probability density function is: ； in, Let be the probability density function of the mathematical bathtub distribution model. For shape distribution parameters, For scale distribution parameters, To normalize the access time interval, It is the inverse hyperbolic tangent function. It is an exponential function with the natural constant e as its base.

5. The computer data center resource scheduling method based on artificial intelligence as described in claim 4, characterized in that, Step S24 specifically includes: Step S241: Based on the observed dataset, take the natural logarithm of the probability density function of the mathematical bathtub distribution model and sum them to construct the log-likelihood function, the mathematical expression of which is: ； in, Let be the log-likelihood function. The total number of samples in the observation dataset. The normalized access time interval for the i-th target data block in the observation dataset; Step S242: Calculate the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter respectively, and combine the first-order partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter to generate a gradient vector; Step S243: Calculate the second partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter respectively, and combine the second partial derivatives of the log-likelihood function with respect to the shape distribution parameter and the scale distribution parameter to generate the Hessian matrix; Step S244: Introduce a fractional logarithmic bound function as a dynamic penalty factor, calculate the iterative update step size, multiply the inverse of the Hessian matrix, the gradient vector, and the fractional logarithmic bound function to calculate and obtain the k-th iterative update step size; Step S245: Using the step size updated in the k-th iteration, perform the (k+1)-th mathematical iteration update on the shape distribution parameters and scale distribution parameters. The mathematical expression for the iterative update is: ； in, To update the step size vector for the k-th iteration, To update the shape distribution parameters of the output for the k-th iteration, To update the scale distribution parameters output in the k-th iteration, To update the shape distribution parameters of the output for the (k+1)th iteration, Update the scale distribution parameters output for the (k+1)th iteration; After each iteration update, the sum of the absolute values ​​of the differences between the current iteration value and the previous iteration value is calculated to obtain the iteration drift.

6. The computer data center resource scheduling method based on artificial intelligence as described in claim 5, characterized in that, Step S244 specifically includes: The calculation process of the fractional-order logarithmic bound function includes: The unconstrained original step size extracted from the product of the inverse of the Hessian matrix and the gradient vector is mathematically expressed as follows: ； in, This represents the original update distance for the shape distribution parameters in the current k-th iteration. This represents the original update distance for the scale distribution parameters in the current k-th iteration, i.e. and This is the unconstrained initial step size; The fractional-order boundary safety factors for shape distribution parameters and scale distribution parameters are calculated independently, and their mathematical expressions are as follows: ； in, The fractional-order boundary safety factor for the shape distribution parameter. The fractional boundary safety factor is the scale distribution parameter. The pre-defined boundary approximation backoff constant; The minimum value between the fractional-order boundary safety factor of the shape distribution parameter and the fractional-order boundary safety factor of the scale distribution parameter is taken as the final output value of the fractional-order logarithmic bound function, and its mathematical expression is: ； in, It is a fractional-order logarithmic bound function; The mathematical expression for calculating the update step size for the k-th iteration is: ； in, It is the inverse of the Hessian matrix. For Hessian matrix, Let k be the gradient vector, and k be the index of the current iteration number.

7. The computer data center resource scheduling method based on artificial intelligence as described in claim 6, characterized in that, Step S3 specifically includes: The pre-defined risk assessment model is a mathematical bathtub distribution risk function, whose mathematical expression is: ； in, Instantaneous risk rate For optimal shape distribution parameters, The optimal scale distribution parameters; The mathematical expression for converting the instantaneous risk rate into the probability of accessing the target within the next unit of time is: ； ； in, This represents the probability of the target being accessed in the next unit of time. This represents the normalized time increment per unit of time. The preset unit time length; Construct an inequality constraint trigger. The decision logic for the inequality constraint trigger is as follows: ； in, The cost of node delays The physical cost of network migration; The cost of node stagnation penalty is the sum of the additional physical time delays required when the next cross-node read event or error correction code verification event occurs if the target data block has not been migrated. The physical cost of network migration is the sum of the transmission bandwidth time and the local flash memory write time required to immediately migrate the target data block to the nearest available node.

8. The computer data center resource scheduling method based on artificial intelligence as described in claim 7, characterized in that, Step S4 includes: When the inequality constraint trigger is set, the target data block is determined to enter the state that must be migrated and a physical migration task is generated. The current bandwidth utilization rate of the network port inside the data center is detected. When the current bandwidth utilization rate is less than the preset idle bandwidth threshold, the physical migration task is immediately executed on the target data block. When the current bandwidth utilization rate is greater than or equal to the preset idle bandwidth threshold, the physical migration task will be placed in the execution queue and its execution will be suspended until the idle bandwidth threshold is met. If the inequality constraint trigger does not hold, the target data block is retained in the current node; Once the physical migration task is initiated, a cumulative probability mapping algorithm based on the bathtub distribution is used to continuously map the normalized access time interval corresponding to the target data block to the standard probability space. The cumulative quantile value of the target data block is then calculated. The calculation of the cumulative quantile value is derived based on the cumulative distribution function of the mathematical bathtub distribution model, and its mathematical expression is as follows: ； in, The cumulative quantile value of the target data block; Send a status polling command to all available candidate nodes in the computer data center to read the current wear index and current read / write queue depth of all available candidate nodes respectively. The current wear index is the number of times the underlying flash memory medium of the available candidate node has been erased and written, and the current read / write queue depth is the total number of input / output requests that are currently being queued and waiting to be processed by the available candidate node. The target access probability is compared with the preset extreme risk threshold, and the cumulative quantile value is compared with the preset very early life cycle threshold and very late life cycle threshold respectively, and a two-dimensional joint node placement strategy is executed.

9. The computer data center resource scheduling method based on artificial intelligence as described in claim 8, characterized in that, The two-dimensional joint node placement strategy specifically includes: If the target access probability is greater than the preset extreme risk threshold and the cumulative quantile value is less than the preset very early life cycle threshold, the target data block is determined to be in the high-frequency writing period at the beginning of its life cycle. Available candidate nodes with current wear indicators less than the preset very low wear safety threshold and current read / write queue depth of zero are extracted as first-level target nodes. The target data block is then forcibly allocated and migrated to the first-level target node. If the target access probability is greater than the preset extreme risk threshold and the cumulative quantile value is greater than the preset late life cycle threshold, the target data block is determined to be in the second burst period before the batch task arrives. The available candidate nodes with the highest read bandwidth index and within the preset wear tolerance range are selected as the second-level target nodes, and the target data block is allocated and migrated to the second-level target nodes. If the above-mentioned criteria for determining the first-level or second-level target node are not met, the target data block is determined to be non-extreme risk data. Nodes within the normal wear range are selected from the candidate nodes as third-level target nodes, and the target data block is allocated and migrated to the third-level target node.

10. An artificial intelligence-based computer data center resource scheduling system, wherein the system is used to execute an artificial intelligence-based computer data center resource scheduling method, characterized in that, It includes a sampling module, a parameter update module, a risk assessment module, and a resource scheduling module: The sampling module is used to collect the identifier and corresponding timestamp of the accessed target data block using a sliding window to obtain time series samples; The parameter update module is used to calculate the physical access time interval from the time series samples and normalize it to obtain the normalized access time interval. Using the normalized access time interval as the observation, a log-likelihood function is constructed based on the maximum likelihood estimation. The Newton-Raphson incremental algorithm is combined with the fractional log bound function to calculate the iterative update step size. The shape distribution parameters and scale distribution parameters are iteratively updated to obtain the optimal shape distribution parameters and the optimal scale distribution parameters. The risk assessment module is used to input the normalized access time interval, optimal shape distribution parameters and optimal scale distribution parameters into the risk assessment model, calculate the instantaneous risk rate and convert it into the target access probability in the next unit time, and construct inequality constraint triggers by combining the node retention penalty cost of the target data block with the physical cost of network migration. The resource scheduling module generates physical migration tasks when the inequality constraint trigger is true, calculates the cumulative quantile value corresponding to the normalized access time interval, and allocates the target data block to the matching target node based on the joint mapping result of the target access probability and the cumulative quantile value.

Images