A method for implementing virtual machine resource isolation for AI training tasks using KubeVirt

Abstract

This invention discloses a method for isolating virtual machine resources for AI training tasks using KubeVirt, relating to the field of virtual machine resource allocation and isolation technology. The method includes the following steps: periodically collecting the utilization rate of each CPU core of each virtual machine in real time using KubeVirt to obtain the raw utilization data of each virtual machine; performing data preprocessing to obtain reference utilization data for each virtual machine and identifying virtual machines that need to be reallocated and isolated; generating reallocation isolation information based on the reference utilization data of the virtual machines requiring reallocation and isolation; and reallocating and isolating the CPU cores of the required virtual machines based on the reallocation isolation information. This invention addresses the problem that existing virtual machine resource allocation and isolation technologies cannot determine the CPU core requirements of virtual machines based on their historical CPU core usage and periodically reallocate and isolate the CPU cores of virtual machines.

Description

Technical Field

[0001] This invention relates to the field of virtual machine resource allocation and isolation technology, specifically a method for implementing virtual machine resource isolation for AI training tasks using KubeVirt. Background Technology

[0002] Virtual machine resource allocation isolation technology is a complete technical system based on hardware-assisted virtualization. It unifies, allocates, and manages the resources of physical servers, such as CPU, memory, storage, network, and acceleration devices, on demand. It ensures that each virtual machine obtains an independent view of hardware resources, and that its allocated resources are not preempted or interfered with by other virtual machines. At the same time, the failure, crash, or malicious behavior of one virtual machine will not spread to other virtual machines on the same physical machine.

[0003] Existing virtual machine resource allocation and isolation technologies often employ permanent binding when allocating and isolating CPU cores for virtual machines used for AI training tasks. This permanently binds the virtual machine's vCPU to a specific physical CPU core, a method that remains unchanged throughout the virtual machine's lifecycle. While this approach eliminates resource contention and context switching overhead, it introduces several problems. AI training tasks exhibit distinct phases, with significant fluctuations in CPU demand throughout their lifecycle, resulting in substantial differences in CPU resource requirements across different stages. Furthermore, under permanent binding, even if certain CPU cores remain underutilized for extended periods, they cannot be used by other tasks or virtual machines, leading to consistently low overall CPU utilization across the cluster. Furthermore, as long as the virtual machine is not deleted, even if there is no load for several consecutive days, these physical cores will be exclusively occupied and cannot be allocated to other tasks or virtual machines that urgently need resources. The CPU load of AI training tasks is highly non-uniform, and usually only a few cores undertake all the actual work, while the rest of the cores are almost completely idle. Permanent binding also locks these idle cores, resulting in a structural waste of some idle cores. Therefore, when allocating and isolating the CPU cores of virtual machines used for AI training tasks, the existing virtual machine resource allocation isolation technology cannot determine the CPU core requirements of the virtual machine based on the historical usage of the virtual machine's CPU cores and periodically reallocate and isolate the virtual machine's CPU cores. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the prior art. It obtains raw utilization data for each virtual machine by periodically and in real-time collecting the utilization rate of each CPU core of each virtual machine using KubeVirt; performs data preprocessing to obtain reference utilization data for each virtual machine, identifies virtual machines requiring reallocation and isolation, generates reallocation and isolation information, and reallocates and isolates the CPU cores of the required virtual machines. This addresses the problem that existing virtual machine resource allocation and isolation technologies, when allocating and isolating CPU cores for virtual machines used for AI training tasks, cannot determine the CPU core requirements of virtual machines based on their historical CPU core usage and periodically reallocate and isolate the CPU cores of virtual machines.

[0005] To achieve the above objectives, this application provides a method for isolating virtual machine resources for AI training tasks using KubeVirt, comprising the following steps:

[0006] KubeVirt periodically collects the utilization rate of each CPU core of each virtual machine in real time to obtain the raw utilization data of each virtual machine.

[0007] The core utilization data of each virtual machine is preprocessed to obtain the reference utilization data of each virtual machine, and the virtual machines that need to be reallocated and isolated are identified.

[0008] Reference data of virtual machines that need to be reallocated and isolated is used to generate reallocation isolation information;

[0009] Based on the reallocation isolation information, the CPU cores of the required virtual machines are reallocated and isolated.

[0010] Furthermore, by periodically collecting the utilization rate of each CPU core of each virtual machine in real time using KubeVirt, the raw utilization data of each virtual machine is obtained through the following sub-steps:

[0011] Obtain all virtual machines used for AI training tasks, and label them from virtual machine 1 to virtual machine n, where n is the total number of virtual machines;

[0012] The CPU cores allocated to virtual machines are denoted as processing cores. For virtual machine 1, the processing cores of virtual machine 1 are denoted as processing core 1 to processing core m, where m is the total number of processing cores of virtual machine 1.

[0013] Furthermore, obtaining the raw utilization data for each virtual machine by periodically collecting the utilization of each CPU core of each virtual machine in real time through KubeVirt includes the following sub-steps:

[0014] Starting from the last time the CPU cores of the virtual machine were reallocated and isolated, the utilization of processing core 1 was collected by KubeVirt at the first time interval, and the collection time was recorded as the utilization information of processing core 1; and the starting collection time was recorded as the cycle start time, where the first time interval was t1.

[0015] Repeatedly and synchronously collect the utilization information of all processing cores of all virtual machines 1 to obtain the raw utilization data of virtual machine 1; and repeatedly and synchronously collect the raw utilization data of all virtual machines.

[0016] Furthermore, the core utilization data of each virtual machine is preprocessed to obtain reference utilization data for each virtual machine, and the virtual machines that need to be reallocated and isolated are identified through the following sub-steps:

[0017] If the time span of the raw utilization data collected from the start of the cycle to the current time is not less than k1, then the utilization information of processing core 1 to processing core m will be arranged in the order of the collection time and recorded as utilization sequence 1 to utilization sequence m, where k1 is the set duration.

[0018] For utilization sequence 1, calculate the corresponding first-order difference sequence, and calculate the median and median absolute deviation MAD of the first-order difference sequence. Obtain the first-order difference in the first-order difference sequence whose absolute difference from the median is greater than 5×MAD, and record it as a candidate step jump point.

[0019] For a candidate step point, based on utilization sequence 1, calculate the average utilization rate of the 10 seconds before and the 10 seconds after the candidate step point. If the difference between the two exceeds k2, it is marked as a real step point. Repeat this process to obtain all real step points. Here, k2 is the set threshold.

[0020] Using the actual step point as the boundary, the utilization sequence 1 is divided into multiple segments, denoted as the load steady-state segment.

[0021] Furthermore, preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated, also includes the following sub-steps:

[0022] Let any steady-state segment of the load be designated as the first steady-state segment. Calculate the coefficient of variation of the first steady-state segment. If the coefficient of variation is <0.1, set the length of the large window to Ae1 and the length of the small window to Ae2. If the coefficient of variation is in the range [0.1, 0.3], set the length of the large window to Be1 and the length of the small window to Be2. If the coefficient of variation is greater than 0.3, set the length of the large window to Ce1 and the length of the small window to Ce2. Where Ae1, Ae2, Be1, Be2, Ce1, and Ce2 are the set window sizes.

[0023] Set the sliding step size to e3, and let the large window and the small window slide backward simultaneously from the starting position of the first steady state segment; according to the 3σ principle, filter out the outliers in the large window and the small window in each sliding step, and denot them as large window outliers and small window outliers respectively; where e3 is the set sliding step size;

[0024] The utilization rate of large window anomalies or small window anomalies in the first steady state segment is recorded as numerical anomaly points; numerical anomaly points in all load steady state segments are repeatedly obtained; numerical anomaly points that are continuous and last for a duration greater than k3 are obtained, and the remaining numerical anomaly points are recorded as candidate anomaly points, where k3 is the set threshold.

[0025] For any candidate anomaly, if at the same acquisition time, the utilization rate of other processing cores is not less than k4% and is also a candidate anomaly, then it is marked as a normal mutation point, and the remaining candidate anomalies are recorded as error points; where k4% is the set threshold.

[0026] Repeatedly obtain all error points in utilization sequence 1 and mark them to obtain effective sequence 1.

[0027] Furthermore, preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated, also includes the following sub-steps:

[0028] The effective sequence 1 is evenly divided into multiple sequence segments of duration k5 according to the acquisition time. Any sequence segment is denoted as the first segment. The total number of utilizations F0 in the first segment is obtained. The proportion of the utilizations of the first segment that are not error points to F0 is calculated and denoted as the effective data rate. Where k5 is the set duration.

[0029] Sequence segments with an effective data rate less than k6 are marked as invalid segments and removed; the weight of utilization rate greater than 70% is set to 3, the weight of utilization rate in the range [30%, 70%] is set to 2, and the weight of utilization rate less than 30% is set to 1, where k6 is the set threshold.

[0030] Calculate the weighted average of the utilization rates of all non-error points in the first sub-segment based on the set weights; record this as the representative utilization rate of the first sub-segment, and record the collection time in the middle of the first sub-segment as the collection time corresponding to the representative utilization rate;

[0031] Repeatedly obtain the representative utilization of all sequence segments and arrange them in chronological order, denoted as the corresponding reference sequence 1; repeatedly obtain the reference sequences corresponding to all processing cores to obtain the reference utilization data of virtual machine 1, and repeatedly obtain the reference utilization data of all virtual machines.

[0032] Furthermore, preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated, also includes the following sub-steps:

[0033] For virtual machine 1, divide reference sequence 1 according to natural weeks to obtain x0 natural weeks. Calculate the 95th percentile and average value of each week, and record them as the weekly peak utilization rate and weekly average utilization rate respectively. Calculate the proportion of the number of utilization rates greater than 70% in each week to the total number of utilization rates in that week, and record it as the weekly high load ratio. After completion, the basic statistical subset of processing core 1 is obtained. Repeat the process of obtaining the basic statistical subset of all processing cores.

[0034] Processing cores with a weekly peak utilization of more than 70% and a weekly high load ratio of more than 20% are designated as high load cores, while processing cores with a weekly peak utilization of less than 20% and a weekly high load ratio of less than 5% are designated as idle cores.

[0035] The proportions of idle cores and high-load cores to the total number of processing cores are calculated weekly and recorded as core redundancy rate and core high load rate, respectively. The difference between the weekly average utilization rate and the weekly average utilization rate of the previous week is calculated and recorded as the load growth trend.

[0036] If virtual machine 1 meets the condition that the core high load rate is less than 30% and the resource redundancy rate is >50% for x0 consecutive weeks, it is marked as resource excess and imbalance.

[0037] If virtual machine 1 has a core load rate greater than 40% for x0 consecutive weeks; or if it has a core load rate greater than 65% for 0.8 x0 weeks in the past x0 weeks, and the load growth trend is greater than 5%, then it is marked as having insufficient resources.

[0038] If virtual machine 1 meets the conditions of resource surplus / imbalance or resource shortage, it is determined to be an isolation to be allocated, and all virtual machines to be allocated are repeatedly obtained.

[0039] Furthermore, generating reallocation isolation information from the reference utilization data of virtual machines that need to be reallocated and isolated as required includes the following sub-steps:

[0040] If virtual machine 1 is to be allocated isolation, for each processing core, calculate the average weekly utilization and the average weekly high load ratio of the past x0 natural weeks, denoted as the average weekly utilization ZP and the average weekly high load ratio ZR; and calculate ZP×ZR, denoted as the corresponding load contribution.

[0041] Calculate the sum of all load contributions, denoted as HD. Accumulate all load contributions from largest to smallest until the sum is not less than 0.95×HD. The processing cores that are accumulated are recorded as effective cores, and the processing cores that are not accumulated are recorded as redundant cores.

[0042] Furthermore, generating reallocation isolation information from the reference utilization data of virtual machines that need to be reallocated and isolated as required also includes the following sub-steps:

[0043] Calculate the average of all weekly peak utilization rates for all effective cores, denoted as effective peak YF; denote the total number of effective cores as GN; calculate ⌈GN×(YF / 70%)⌉, denoted as the basic demand core count AE;

[0044] Set the safety margin to AW, calculate the standard deviation of the weekly average utilization rate of each effective core, and then calculate the mean of all standard deviations, denoted as load volatility FB; if FB < 10%, then set AW = 1; if 10% ≤ FB ≤ 30%, then set AW = 2; if FB > 30%, then set AW = 3.

[0045] Calculate AE+AW, denoted as the final core requirement AU for virtual machine 1; repeatedly obtain the final core requirement of all virtual machines to be allocated isolation, denoted as the reallocation isolation information.

[0046] Furthermore, based on the reallocation isolation information, the reallocation and isolation of the CPU cores of the required virtual machines includes the following sub-steps:

[0047] If virtual machine 1 is to be allocated isolation, calculate ⌊0.5×m⌋ and record it as the peak value of increase or decrease; based on the final core requirement of virtual machine 1; under the premise that the increase or decrease in the number of CPU cores does not exceed ⌊0.5×m⌋, use KubeVirt to reallocate the CPU cores of virtual machine 1 and isolate them, so that the number of CPU cores of virtual machine 1 after reallocation is closest to or equal to AU.

[0048] Based on the final core requirement of the virtual machines to be isolated, repeat the reallocation and isolation process for all virtual machines to be isolated.

[0049] The beneficial effects of this invention are as follows: This invention periodically collects the utilization rate of each CPU core of each virtual machine in real time using KubeVirt to obtain the raw utilization data of each virtual machine; it preprocesses the core utilization data of each virtual machine to obtain the reference utilization data of each virtual machine and identifies the virtual machines that need to be reallocated and isolated; it generates reallocation isolation information based on the reference utilization data of the virtual machines that need to be reallocated and isolated; and it reallocates and isolates the CPU cores of the virtual machines that need to be reallocated and isolated based on the reallocation isolation information. When reallocating and isolating the CPU cores of virtual machines used for AI training tasks, the CPU core requirements of virtual machines can be determined according to the historical usage of the CPU cores of virtual machines, and the CPU cores of virtual machines can be reallocated and isolated periodically to reduce the waste of CPU resources.

[0050] This invention first identifies step points and divides steady-state segments in the original utilization data, and then uses a dual-window approach (large and small windows) for anomaly screening. This process gradually eliminates interference data caused by stage switching, instantaneous jitter, acquisition noise, and occasional spikes during training, retaining only reference utilization data that better reflects the actual load level of the virtual machine, thus ensuring the accuracy of subsequent identification. By processing the weekly average utilization rate and weekly high load ratio of cores over several natural weeks for comprehensive calculation, and calculating the load contribution, effective cores are selected by accumulating from high to low load contribution. This highlights cores that contribute significantly to the actual training task of the virtual machine. Furthermore, by combining the average weekly peak utilization rate of effective cores, the number of cores, and the load volatility to introduce the basic required number of cores and safety margin, the final required number of cores reflects both the actual peak demand and the redundancy buffer caused by training load fluctuations. This improves the accuracy and robustness of the required core estimation and reduces waste caused by excessive CPU core allocation. Attached Figure Description

[0051] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0052] Figure 2 This is a flowchart of the error point screening process of the present invention;

[0053] Figure 3 This is a flowchart of the isolation determination process of the present invention;

[0054] Figure 4 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1, please refer to Figure 1 As shown, this application provides a method for isolating virtual machine resources for AI training tasks using KubeVirt, including the following steps:

[0057] Step S1 involves periodically collecting the utilization rate of each CPU core of each virtual machine in real time using KubeVirt to obtain the raw utilization data for each virtual machine. Step S1 includes the following sub-steps:

[0058] Step S101: Obtain all virtual machines used for AI training tasks, and label them as virtual machine 1 to virtual machine n, where n is the total number of virtual machines;

[0059] Step S102: The CPU cores allocated to the virtual machine are denoted as processing cores. For virtual machine 1, the processing cores of virtual machine 1 are denoted as processing core 1 to processing core m in sequence, where m is the total number of processing cores of virtual machine 1.

[0060] Step S103: Starting from the last completion of the CPU core reallocation and isolation of the virtual machine, the utilization rate of processing core 1 is collected by KubeVirt at the first time interval, and the collection time is recorded as the utilization rate information of processing core 1; and the starting collection time is recorded as the cycle start time, where the first time interval is t1; in this embodiment, t1 = 1 second, that is, 1 time per second, which can be flexibly set, generally [1 second, 5 seconds];

[0061] KubeVirt is an open-source cloud-native virtualization project that extends the Kubernetes API and controller mechanism to incorporate virtual machines into the unified orchestration system of Kubernetes, enabling hybrid deployment and unified management of container and virtual machine workloads. KubeVirt natively supports collecting runtime status data for each vCPU, which can calculate core utilization.

[0062] Step S104: Repeatedly and synchronously collect the utilization information of all processing cores of all virtual machines 1 to obtain the raw utilization data of virtual machine 1; and repeatedly and synchronously collect the raw utilization data of all virtual machines.

[0063] In the specific implementation process, since AI training tasks usually have the characteristics of uneven core utilization, with a few cores under high load and others idle, it is necessary to collect the utilization rate of each processing core. This can more accurately capture the load difference and provide a basis for subsequent identification of high-load cores, idle cores and redundant cores, thereby avoiding the resource usage situation being masked by the overall average utilization rate of the virtual machine.

[0064] Step S2 involves preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated. Step S2 includes the following sub-steps:

[0065] Step S201: If the time span of the collected raw utilization data from the start of the cycle to the current time is not less than k1, then the utilization information of processing core 1 to processing core m is arranged according to the order of the collection time and recorded as utilization sequence 1 to utilization sequence m, where k1 is the set duration; in this embodiment, k1 = 1 month, which can be flexibly set. k1 avoids unstable judgment due to too few samples and avoids frequent reallocation and isolation, which affects the AI ​​training task.

[0066] For step S202, please refer to... Figure 2 As shown, for utilization sequence 1, the corresponding first-order difference sequence is calculated, and the median and median absolute deviation MAD of the first-order difference sequence are calculated. The first-order difference in the first-order difference sequence with an absolute difference greater than 5×MAD from the median is recorded as a candidate step point.

[0067] Because short-term jitter, spikes and noise are common in AI training tasks, if we judge directly based on the load level, it is easy to misjudge normal fluctuations as load switching. However, using the median absolute deviation as a benchmark can reduce the impact of extreme values ​​on the judgment results, more accurately capture the location of phased load changes, screen out the points that may truly represent training phase switching and sudden changes in resource demand, and reduce misidentification caused by random noise.

[0068] Step S203: For candidate step points, based on utilization sequence 1, calculate the average utilization rate of the 10 seconds before and the 10 seconds after the candidate step point. If the difference between the two exceeds k2, it is marked as a real step point, and all real step points are repeatedly acquired. Here, k2 is a set threshold. In this embodiment, k2=30%, which can be set flexibly, generally [25%, 35%]. k2 is used to distinguish between instantaneous acquisition spikes and real load steps. After a real load step, the load will stabilize at the new level.

[0069] Step S204: Using the actual step point as the boundary, that is, using the midpoint between the two utilization rates corresponding to the actual step point as the boundary, divide the utilization rate sequence 1 into multiple segments, denoted as the load steady-state segment.

[0070] The entire utilization sequence is divided into several relatively stable segments according to the actual step point; this is equivalent to first separating different load stages in the AI ​​training task, and then filtering out outliers based on the data features within each segment, thus avoiding mixing data with different features and causing filtering errors.

[0071] Step S205: Designate any steady-state load segment as the first steady-state segment, calculate the coefficient of variation of the first steady-state segment. If the coefficient of variation < 0.1, set the length of the large window to Ae1 and the length of the small window to Ae2; if the coefficient of variation is within [0.1, 0.3], set the length of the large window to Be1 and the length of the small window to Be2; if the coefficient of variation is greater than 0.3, set the length of the large window to Ce1 and the length of the small window to Ce2. Wherein, Ae1, Ae2, Be1, Be2, Ce1, and Ce2 are the set window sizes. In this embodiment, Ae1, Ae2, Be1, Be2, Ce1, and Ce2 are in units of duration: Ae1 = 30 seconds, Ae2 = 15 seconds, Be1 = 30 seconds, Be2 = 10 seconds, Ce1 = 15 seconds, and Ce2 = 5 seconds. These values ​​can be flexibly set.

[0072] The anomaly detection window is adaptively adjusted based on the degree of fluctuation within the steady-state segment. The larger the coefficient of variation, the stronger the fluctuation within the segment, and the more careful the window setting needs to be, i.e., a smaller window. The smaller the coefficient of variation, the more stable the segment, and a larger window can be used to reduce computation, avoid the one-size-fits-all problem caused by using a fixed window, and improve the detection effect of steady-state segments with different fluctuations.

[0073] Step S206: Set the sliding step size to e3, and let the large window and the small window slide backward simultaneously from the starting position of the first steady state segment; according to the 3σ principle, filter the outliers in the large window and the small window in each sliding, and record them as large window outliers and small window outliers respectively; where e3 is the set sliding step size; in this embodiment, the sliding step size is 2, that is, two data points are slid in each step, which can be flexibly set.

[0074] Step S207: Record the utilization rate of large window anomaly or small window anomaly in the first steady state segment as numerical anomaly points; repeatedly acquire numerical anomaly points in all load steady state segments; acquire continuous numerical anomaly points with a duration greater than k3, and record the remaining numerical anomaly points as candidate anomaly points, where k3 is the set threshold; in this embodiment, k3=4s, which can be flexibly set according to t1, generally 3 to 5 acquisition times;

[0075] Larger windows are better suited for grasping overall trends, while smaller windows are better suited for identifying local spikes. Combining the two can take into account both the global and local aspects, improving the reliability and accuracy of anomaly detection and reducing missed detections caused by a single window scale. Continuous anomalies over a longer period of time are more likely to be genuine load fluctuations rather than sampling errors.

[0076] Step S208: For any candidate anomaly, if the utilization rate of other processing cores at the same acquisition time is not less than k4%, it is also a candidate anomaly. Then it is marked as a normal mutation point, and the remaining candidate anomalies are recorded as error points. Here, k4% is the set threshold. In this embodiment, k4%=50%, which can be set flexibly. Generally, it is [40%, 30%].

[0077] For example, if the first steady-state segment is [30, 30, 30, 30, 30, 98, 30, 30, 30, 30, 40, 40, 40, 40, 40, 30, 30, 30, 30, 30], and the unit symbol is %, then the coefficient of variation is 15.7%. Therefore, the large window Be1 = 40 seconds. Since the first steady-state segment is only 20 seconds, the large window is reduced to 20 seconds. The small window Be2 = 15 seconds. 98% of these are large window anomalies, which are numerical anomalies, and since they are isolated numerical anomalies, they are candidate anomalies. Virtual machine 1 has 6 processing cores. At the same acquisition time, if the utilization rate of the other 5 processing cores is not a candidate anomaly, then 98% are error points.

[0078] Step S209: Repeatedly obtain all error points in utilization sequence 1 and mark them to obtain effective sequence 1.

[0079] Step S210: Divide the valid sequence 1 evenly into multiple sequence segments of duration k5 according to the acquisition time, and denote any sequence segment as the first segment; obtain the total number of utilization points F0 in the first segment, and calculate the proportion of the number of utilization points in the first segment that are not error points to F0, which is denoted as the effective data rate; where k5 is the set duration; in this embodiment, k5 = 1 minute, which can be flexibly set, generally [30 seconds, 2 minutes]; dividing into sequence segments is to reduce the time granularity and improve the calculation efficiency; the effective data rate is used to quantify the credibility of each sequence segment; sequence segments with too low an effective data rate have too low reference value and can be reduced in weight or eliminated in subsequent decisions;

[0080] Step S211: Mark the sequence segments with an effective data rate less than k6 as invalid segments and remove them; set the weight of the utilization rate greater than 70% to 3, set the weight of the utilization rate in the range of [30%, 70%] to 2, and set the weight of the utilization rate less than 30% to 1; where k6 is the set threshold. In this embodiment, k6=30%, which can be set flexibly, generally [20%, 40%], and the weights corresponding to different utilization rates can also be set flexibly.

[0081] Step S212: Calculate the weighted average of the utilization rates of all non-error points in the first sub-segment according to the set weights; record it as the representative utilization rate of the first sub-segment, and record the collection time in the middle of the first sub-segment as the collection time corresponding to the representative utilization rate; by assigning weights to segments, the contribution of higher utilization rates to the results is highlighted, because in AI training tasks, high utilization rates usually better reflect the real resource demand, while low utilization rates reflect more idle or light load states, and compared with ordinary average values, they better reflect the real resource demand in that time period.

[0082] For example, let Vi be the utilization rate in the first sub-segment, where i is the position number, V1 to V20 is 20%, V21 to V31 is 60%, V32 is the error point, V33 to V40 is 60%, V41 to V50 is 80%, and V51 to V60 is 60%.

[0083] There are 20 instances of utilization rates less than 30% at 20%, with a weight of 1, totaling 20 × 20% × 1 = 400%; there are 29 instances of utilization rates between 30% and 70%, at 60%, with a weight of 2, totaling 29 × 60% × 2 = 3480%; there are 10 instances of utilization rates greater than 70%, at 80%, with a weight of 3, totaling 10 × 80% × 3 = 2400%; the total weight is 20 + 2 × 29 + 3 × 10 = 108; therefore, the representative utilization rate of the first sub-segment is (400% + 3480% + 2400%) / 108 = 62.8%;

[0084] Step S213: Repeatedly obtain the representative utilization rate of all sequence segments and arrange them in chronological order, denoted as the corresponding reference sequence 1; repeatedly obtain the reference sequence corresponding to all processing cores to obtain the reference utilization data of virtual machine 1, and repeatedly obtain the reference utilization data of all virtual machines.

[0085] For step S214, please refer to... Figure 3 As shown, for virtual machine 1, reference sequence 1 is divided into x0 natural weeks. The 95th percentile and average value of each week are calculated and denoted as the weekly peak utilization rate and weekly average utilization rate, respectively. The proportion of the number of utilization rates greater than 70% in each week to the total number of utilization rates in that week is calculated and denoted as the weekly high load ratio. After completion, the basic statistical subset of processing core 1 is obtained. The basic statistical subset of all processing cores is obtained repeatedly. The weekly peak utilization rate reflects the high resource demand, the weekly average utilization rate reflects the overall resource demand, and the weekly high load ratio reflects the frequency of the continuous occurrence of high resource demand.

[0086] Step S215: Processing cores with a weekly peak utilization rate greater than 70% and a weekly high load ratio greater than 20% are recorded as high load cores, and processing cores with a weekly peak utilization rate less than 20% and a weekly high load ratio less than 5% are recorded as idle cores.

[0087] Step S216: Calculate the proportion of idle cores and high-load cores to the total number of processing cores each week, and record them as core redundancy rate and core high load rate respectively; calculate the difference between the weekly average utilization rate and the weekly average utilization rate of the previous week, and record it as the load growth trend; core redundancy rate and high load rate reflect resource structure, and load growth trend reflects changes in future resource demand, which is used to determine whether resource demand is rising or falling.

[0088] Step S217: If virtual machine 1 meets the condition that the core high load rate is less than 30% and the resource redundancy rate is >50% for x0 consecutive weeks, it is marked as resource excess imbalance.

[0089] Step S218: If virtual machine 1 meets the core high load rate of more than 40% for x0 consecutive weeks; or if ⌊0.8×x0⌋ weeks in the past x0 weeks meet the core high load rate of more than 65% and the load growth trend is greater than 5%, then it is marked as resource shortage; the judgment thresholds corresponding to core high load rate, resource redundancy rate and load growth trend can be flexibly adjusted according to the actual application scenario.

[0090] Step S219: If virtual machine 1 meets the conditions of resource surplus / imbalance or resource shortage, it is determined to be an isolation to be allocated, and all virtual machines to be allocated are repeatedly obtained.

[0091] In practice, only virtual machines that exhibit unreasonable resource conditions can be reallocated to avoid indiscriminate adjustments to all virtual machines, reduce unnecessary resource migration and isolation operations, minimize system disturbances, and make resource adjustments more targeted.

[0092] Step S3: Generate reallocation isolation information based on the reference utilization data of the virtual machines to be reallocated and isolated as needed; Step S3 includes the following sub-steps:

[0093] Step S301: If virtual machine 1 is to be allocated isolation, for each processing core, calculate the average weekly utilization rate and the average weekly high load ratio of the past x0 natural weeks, denoted as the average weekly utilization rate ZP and the high load ratio ZR; and calculate ZP×ZR, denoted as the corresponding load contribution; that is, obtain the load contribution of each processing core; ZP reflects the average resource consumption level of the core in the whole cycle, and ZR reflects the frequency of the core being in a high load state. The load contribution obtained by multiplying the two is equivalent to combining load intensity and load persistence to evaluate the actual contribution of the core to the virtual machine training task.

[0094] Step S302: Calculate the sum of all load contributions, denoted as HD. Accumulate all load contributions from largest to smallest until the sum is not less than 0.95×HD. The processing cores that are accumulated are recorded as effective cores, and the processing cores that are not accumulated are recorded as redundant cores. This can more accurately identify the cores that are actually undertaking the main computing tasks.

[0095] Step S303: Calculate the average of all weekly peak utilization rates of all effective cores, denoted as effective peak value YF; denote the total number of effective cores as GN; calculate ⌈GN×(YF / 70%)⌉, denoted as the basic demand core count AE; YF reflects the resource demand of effective cores at the peak level; the basic demand core count reflects the actual peak load demand, reducing the risk of insufficient or excessive core allocation; where 70% is to reserve 30% elasticity space to cope with sudden load fluctuations and data loading peaks, while not reserving too many idle resources.

[0096] Step S304: Set the safety margin to AW, calculate the standard deviation of the weekly average utilization rate of each effective core, and then calculate the mean of all standard deviations, denoted as the load volatility FB; if FB < 10%, then set AW = 1; if 10% ≤ FB ≤ 30%, then set AW = 2; if FB > 30%, then set AW = 3; the safety margin corresponding to different load volatility can be flexibly adjusted according to the actual application scenario.

[0097] Step S305: Calculate AE+AW, and record it as the final core requirement AU of virtual machine 1; repeatedly obtain the final core requirement of all virtual machines to be allocated isolation, and record it as the reallocation isolation information.

[0098] For example, virtual machine 1 has 6 processing cores. The average cycle utilization of processing core 1 is ZP=65.5%, and the cycle high load ratio is ZR=45.25%. Therefore, the load contribution ZP×ZR=65.5%×45.25%=29.64%. Repeating the calculation of the load contribution of all processing cores, the load contribution of processing cores 1 to 6 is [29.640%, 26.560%, 2.150%, 0.038%, 0.008%, 0%], respectively. The sum of the load contributions HD=58.396%. Adding 29.640% and 26.560% together, we get 56.2%, which accounts for 96.24% of HD. Therefore, processing cores 1 and 2 are effective cores, and the rest are redundant cores.

[0099] Calculate the mean of the peak utilization of all effective cores for all weeks, and we get the effective peak value YF = 90.25% and GN = 2. Then the basic demand cores AE = ⌈GN × (YF / 70%)⌉ = ⌈2 × (90.25% / 70%)⌉ = 3. Calculate the standard deviation of the weekly average utilization of processing core 1 and processing core 2, which are 2.29% and 1.80% respectively. Then the load fluctuation rate FB = (2.29% + 1.80%) / 2 = 2.05%, which is less than 10%. Therefore, the safety margin AW = 1. So the final demand cores AU = AE + AW = 4.

[0100] In practice, setting a safety margin based on load volatility means dynamically adjusting the size of the safety margin according to the dynamic characteristics of the task load, avoiding a one-size-fits-all approach. If the same safety margin is set for all tasks, either insufficient margin will lead to an immediate bottleneck after adjustment, or excessive margin will cause new resource waste. The more stable the load, the less margin is reserved; the more volatile the load, the more margin is reserved. This ensures that the adjusted configuration can cope with possible future load growth and fluctuations, while minimizing unnecessary resource reservations and further improving resource utilization.

[0101] Step S4: Based on the reallocation isolation information, reallocate and isolate the CPU cores of the required virtual machines; Step S4 includes the following sub-steps:

[0102] Step S401: If virtual machine 1 is to be allocated isolation, calculate ⌊0.5×m⌋ and record it as the peak value of increase or decrease; based on the final core requirement of virtual machine 1; under the premise that the increase or decrease in the number of CPU cores does not exceed ⌊0.5×m⌋, avoid excessive adjustment to avoid task performance fluctuations; use KubeVirt to reallocate the CPU cores of virtual machine 1 and isolate them, so that the number of CPU cores of virtual machine 1 after reallocation is closest to or equal to AU;

[0103] Step S402: Based on the final core requirement of the virtual machines to be isolated, repeat the reallocation and isolation process for all virtual machines to be isolated.

[0104] In the specific implementation process, virtual machine CPU cores are periodically reallocated and isolated based on historical utilization data; this ensures the computing resources required for training tasks to run, while releasing redundant cores for other tasks as much as possible, thereby improving the overall utilization of physical server CPUs, reducing resource idleness, and improving resource scheduling efficiency.

[0105] Example 2, please refer to Figure 4 As shown, Figure 4A schematic diagram of an electronic device is provided, which may include a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The memory stores computer-readable instructions, and the processor can call these instructions. When the processor executes a computer-readable instruction, it performs steps such as those in the method for isolating virtual machine resources for AI training tasks using KubeVirt, to achieve the following functions: periodically and in real-time collecting the utilization rate of each CPU core of each virtual machine using KubeVirt to obtain the raw utilization data of each virtual machine; preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated; generating reallocation isolation information based on the reference utilization data of the virtual machines that need to be reallocated and isolated; and reallocating and isolating the CPU cores of the required virtual machines based on the reallocation isolation information.

[0106] Furthermore, when the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] Example 3: This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps of a method for isolating virtual machine resources for AI training tasks using KubeVirt, to achieve the following functions: periodically and in real-time collecting the utilization rate of each CPU core of each virtual machine using KubeVirt to obtain the raw utilization data of each virtual machine; performing data preprocessing on the core utilization data of each virtual machine to obtain reference utilization data of each virtual machine, and obtaining the virtual machines that need to be reallocated and isolated; generating reallocation isolation information based on the reference utilization data of the virtual machines that need to be reallocated and isolated; and reallocating and isolating the CPU cores of the virtual machines that need to be reallocated and isolated based on the reallocation isolation information.

[0108] Based on the above description of the embodiments, the embodiments of the present invention can be provided as methods, systems, or computer program products. Based on this understanding, the above technical solutions, in essence or in terms of their contribution to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or certain parts of the embodiments.

[0109] In the embodiments provided in this application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces. The indirect coupling or communication connection between systems, modules, and units may be electrical, mechanical, or other forms.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for isolating virtual machine resources for AI training tasks using KubeVirt, characterized in that, Includes the following steps: KubeVirt periodically collects the utilization rate of each CPU core of each virtual machine in real time to obtain the raw utilization data of each virtual machine. The core utilization data of each virtual machine is preprocessed to obtain the reference utilization data of each virtual machine, and the virtual machines that need to be reallocated and isolated are identified. Reference data of virtual machines that need to be reallocated and isolated is used to generate reallocation isolation information; Based on the reallocation isolation information, the CPU cores of the required virtual machines are reallocated and isolated.

2. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 1, characterized in that, The raw utilization data for each virtual machine is obtained by periodically collecting the utilization of each CPU core of each virtual machine in real time using KubeVirt, which includes the following sub-steps: Obtain all virtual machines used for AI training tasks, and label them from virtual machine 1 to virtual machine n, where n is the total number of virtual machines; The CPU cores allocated to virtual machines are denoted as processing cores. For virtual machine 1, the processing cores of virtual machine 1 are denoted as processing core 1 to processing core m, where m is the total number of processing cores of virtual machine 1.

3. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 2, characterized in that, The process of periodically collecting the utilization rate of each CPU core of each virtual machine in real time using KubeVirt to obtain the raw utilization data of each virtual machine also includes the following sub-steps: Starting from the last time the CPU cores of the virtual machine were reallocated and isolated, the utilization of processing core 1 was collected by KubeVirt at the first time interval, and the collection time was recorded as the utilization information of processing core 1; and the starting collection time was recorded as the cycle start time, where the first time interval was t1. Repeatedly and synchronously collect the utilization information of all processing cores of all virtual machines 1 to obtain the raw utilization data of virtual machine 1; and repeatedly and synchronously collect the raw utilization data of all virtual machines.

4. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 3, characterized in that, The process of preprocessing the core utilization data for each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated, includes the following sub-steps: If the time span of the raw utilization data collected from the start of the cycle to the current time is not less than k1, then the utilization information of processing core 1 to processing core m will be arranged in the order of the collection time and recorded as utilization sequence 1 to utilization sequence m, where k1 is the set duration. For utilization sequence 1, calculate the corresponding first-order difference sequence, and calculate the median and median absolute deviation MAD of the first-order difference sequence. Obtain the first-order difference in the first-order difference sequence whose absolute difference from the median is greater than 5×MAD, and record it as a candidate step jump point. For a candidate step point, based on utilization sequence 1, calculate the average utilization rate of the 10 seconds before and the 10 seconds after the candidate step point. If the difference between the two exceeds k2, it is marked as a real step point. Repeat this process to obtain all real step points. Here, k2 is the set threshold. Using the actual step point as the boundary, the utilization sequence 1 is divided into multiple segments, denoted as the load steady-state segment.

5. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 4, characterized in that, The process of preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated, also includes the following sub-steps: Let any steady-state segment of the load be designated as the first steady-state segment. Calculate the coefficient of variation of the first steady-state segment. If the coefficient of variation is <0.1, set the length of the large window to Ae1 and the length of the small window to Ae2. If the coefficient of variation is in the range [0.1, 0.3], set the length of the large window to Be1 and the length of the small window to Be2. If the coefficient of variation is greater than 0.3, set the length of the large window to Ce1 and the length of the small window to Ce2. Where Ae1, Ae2, Be1, Be2, Ce1, and Ce2 are the set window sizes. Set the sliding step size to e3, and let the large window and the small window slide backward simultaneously from the starting position of the first steady state segment; according to the 3σ principle, filter out the outliers in the large window and the small window in each sliding step, and denot them as large window outliers and small window outliers respectively; where e3 is the set sliding step size; The utilization rate of large window anomalies or small window anomalies in the first steady state segment is recorded as numerical anomaly points; numerical anomaly points in all load steady state segments are repeatedly obtained; numerical anomaly points that are continuous and last for a duration greater than k3 are obtained, and the remaining numerical anomaly points are recorded as candidate anomaly points, where k3 is the set threshold. For any candidate anomaly, if at the same acquisition time, the utilization rate of other processing cores is not less than k4% and is also a candidate anomaly, then it is marked as a normal mutation point, and the remaining candidate anomalies are recorded as error points; where k4% is the set threshold. Repeatedly obtain all error points in utilization sequence 1 and mark them to obtain effective sequence 1.

6. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 5, characterized in that, The process of preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated, also includes the following sub-steps: The effective sequence 1 is evenly divided into multiple sequence segments of duration k5 according to the acquisition time. Any sequence segment is denoted as the first segment. The total number of utilizations F0 in the first segment is obtained. The proportion of the utilizations of the first segment that are not error points to F0 is calculated and denoted as the effective data rate. Here, k5 is the set duration. Sequence segments with an effective data rate less than k6 are marked as invalid segments and removed; the weight of utilization rate greater than 70% is set to 3, the weight of utilization rate in the range [30%, 70%] is set to 2, and the weight of utilization rate less than 30% is set to 1, where k6 is the set threshold. Calculate the weighted average of the utilization rates of all non-error points in the first sub-segment based on the set weights; record this as the representative utilization rate of the first sub-segment, and record the collection time in the middle of the first sub-segment as the collection time corresponding to the representative utilization rate; Repeatedly obtain the representative utilization of all sequence segments and arrange them in chronological order, denoted as the corresponding reference sequence 1; Repeatedly obtain the reference sequences corresponding to all processing cores to obtain the reference utilization data of virtual machine 1, and repeat the acquisition of reference utilization data of all virtual machines.

7. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 6, characterized in that, The process of preprocessing the core utilization data of each virtual machine to obtain reference utilization data for each virtual machine, and identifying the virtual machines that need to be reallocated and isolated, also includes the following sub-steps: For virtual machine 1, divide reference sequence 1 according to natural weeks to obtain x0 natural weeks. Calculate the 95th percentile and average value of each week, and record them as the weekly peak utilization rate and weekly average utilization rate respectively. Calculate the proportion of the number of utilization rates greater than 70% in each week to the total number of utilization rates in that week, and record it as the weekly high load ratio. Upon completion, the basic statistical subset of processing core 1 is obtained; Repeatedly retrieve the basic statistical subsets of all core processing functions; Processing cores with a peak weekly utilization of more than 70% and a high weekly load ratio of more than 20% are designated as high load cores, while processing cores with a peak weekly utilization of less than 20% and a high weekly load ratio of less than 5% are designated as idle cores. The proportions of idle cores and high-load cores to the total number of processing cores are calculated separately each week and recorded as core redundancy rate and core high load rate, respectively. The difference between the weekly average utilization rate and the weekly average utilization rate of the previous week is calculated and recorded as the load growth trend. If virtual machine 1 meets the conditions of core high load rate less than 30% and resource redundancy rate > 50% for x0 consecutive weeks, it is marked as resource excess imbalance. If virtual machine 1 has a core load rate greater than 40% for x0 consecutive weeks; or if it has a core load rate greater than 65% for 0.8 x0 weeks in the past x0 weeks, and the load growth trend is greater than 5%, then it is marked as having insufficient resources. If virtual machine 1 meets the conditions of resource surplus / imbalance or resource shortage, it is determined to be an isolation to be allocated, and all virtual machines to be allocated are repeatedly obtained.

8. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 7, characterized in that, Generating reallocation isolation information from reference data of virtual machines that need to be reallocated and isolated includes the following sub-steps: If virtual machine 1 is to be allocated isolation, for each processing core, calculate the average weekly utilization and the average weekly high load ratio of the past x0 natural weeks, denoted as the average weekly utilization ZP and the average weekly high load ratio ZR; and calculate ZP×ZR, denoted as the corresponding load contribution. Calculate the sum of all load contributions, denoted as HD. Accumulate all load contributions from largest to smallest until the sum is not less than 0.95×HD. The processing cores that are accumulated are recorded as effective cores, and the processing cores that are not accumulated are recorded as redundant cores.

9. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 8, characterized in that, Generating reallocation isolation information from reference data of virtual machines that need to be reallocated and isolated as required also includes the following sub-steps: Calculate the average of all weekly peak utilization rates for all effective cores, denoted as effective peak YF; denote the total number of effective cores as GN; calculate ⌈GN×(YF / 70%)⌉, denoted as the basic demand core count AE; Set the safety margin to AW, calculate the standard deviation of the weekly average utilization rate of each effective core, and then calculate the mean of all standard deviations, denoted as load volatility FB; if FB < 10%, then set AW = 1; if 10% ≤ FB ≤ 30%, then set AW = 2; if FB > 30%, then set AW = 3. Calculate AE+AW, denoted as the final core requirement AU for virtual machine 1; repeatedly obtain the final core requirement of all virtual machines to be allocated isolation, denoted as the reallocation isolation information.

10. The method for isolating virtual machine resources for AI training tasks using KubeVirt according to claim 9, characterized in that, Based on the reallocation isolation information, the reallocation and isolation of the CPU cores of the required virtual machines includes the following sub-steps: If virtual machine 1 is to be allocated isolation, calculate ⌊0.5×m⌋ and record it as the peak value of increase or decrease; based on the final core requirement of virtual machine 1; under the premise that the increase or decrease in the number of CPU cores does not exceed ⌊0.5×m⌋, use KubeVirt to reallocate the CPU cores of virtual machine 1 and isolate them, so that the number of CPU cores of virtual machine 1 after reallocation is closest to or equal to AU. Based on the final core requirement of the virtual machines to be isolated, repeat the reallocation and isolation process for all virtual machines to be isolated.

Images