A model training resource isolation and management system and method in a multi-tenant environment

Abstract

The present invention relates to the field of cloud computing resource management technology, and in particular to a model training resource isolation and management system and method in a multi-tenant environment, the system comprising a data acquisition module, a resource adjustment module, a trend prediction module, an evaluation adjustment module, and an execution module. The present invention monitors the resource usage of each named isolated tenant in the model training process in real time, combines preset evaluation criteria and trend prediction, dynamically adjusts resource quotas, ensures reasonable allocation and efficient use of resources, and through comprehensive evaluation of priority and synchronization, the system can timely correct resource allocation strategies when resource usage fluctuates, avoid resource waste or excessive occupation, and improve the resource management efficiency of the overall system. This dynamic allocation mechanism not only improves resource utilization, but also adapts to changes in demand for different tenants, effectively solving the problem of low resource utilization efficiency and unstable system performance due to reliance on static configuration.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing resource management, and in particular to a model training resource isolation and management system and method in a multi-tenant environment. Background Art

[0002] In the current field of machine learning and artificial intelligence, multiple teams and users often need to share the same computing cluster for model training. However, this shared environment often faces problems such as resource competition, chaotic permission management, and environmental pollution, which seriously affects the stability and efficiency of model training. It may also lead to security risks such as data leakage and permission abuse. Traditional resource management methods often make it difficult to effectively isolate the environments of different tenants and perform refined permission control while ensuring full utilization of resources.

[0003] The patent document with publication number CN115686843A discloses a heterogeneous virtualization multi-tenant resource isolation scheduling method, which includes: an administrator adds a filter; an administrator creates a tenant and creates a tenant identifier for the tenant; the tenant creates a corresponding tenant group and a tenant template, and compiles the filtering rules of the filter corresponding to the tenant group according to the isolation requirements; the tenant issues a request to create a virtual machine, and the filter determines whether the tenant is creating a virtual machine for the first time. If so, according to the tenant template requirements, the virtual machine is deployed on the physical host with the largest free memory resources; if not, the filter gives priority to the physical host that already has the tenant and determines whether the physical host meets the filtering rules. If so, according to the tenant template requirements, the virtual machine is deployed on the physical host. It can be seen that this method relies on preset rules and templates for resource allocation, cannot respond to changes in resource demand in a timely manner, and rarely involves the prediction of resource usage trends and the dynamic adjustment of priorities. It lacks flexibility in resource contention, focuses on the execution of initial allocation rules when allocating resources, and does not fully consider the non-optimized state that may occur during resource use, and cannot meet the multi-tenant resource management needs in complex environments. Summary of the invention

[0004] To this end, the present invention provides a model training resource isolation and management system and method in a multi-tenant environment, so as to overcome the problems in the prior art of low resource utilization efficiency and unstable system performance due to reliance on static configuration.

[0005] To achieve the above objectives, the present invention provides a model training resource isolation and management system in a multi-tenant environment, including:

[0006] The data collection module is used to monitor the CPU usage and memory usage of each named isolated tenant during the model training process in real time;

[0007] A resource adjustment module, connected to the data acquisition module, comprising a calculation and determination unit and an adjustment unit, wherein the calculation and determination is used to calculate a resource usage evaluation value according to a CPU usage rate and a memory usage rate, and to make a determination according to the resource usage evaluation value and a preset standard evaluation value to form a first determination result, and the adjustment unit is used to adjust a preset resource quota of each named isolation tenant according to the first determination result to form an adjusted resource quota;

[0008] a trend prediction module, connected to the data acquisition module and the resource adjustment module respectively, for determining the change trend of the total resource usage evaluation value within the preset prediction time according to the first judgment result, the total resource usage evaluation value within the preset prediction time and the preset standard resource usage change rate, and making a judgment according to the change trend, the total resource usage evaluation value within the preset prediction time and the preset evaluation value standard deviation threshold to form a second judgment result;

[0009] An evaluation and adjustment module, connected to the data collection module and the trend prediction module respectively, for determining the priority of each named isolation tenant according to the second determination result, the service level of each named isolation tenant and the preset service level agreement, calculating the synchronization degree according to the priority, the total CPU usage rate and the total memory usage rate within the preset prediction time, and adjusting the preset resource quota of each named isolation tenant according to the synchronization degree and the preset standard synchronization degree range to form an adjusted resource quota;

[0010] The execution module is connected to the resource adjustment module and the evaluation adjustment module respectively, and is used to perform resource allocation for each named isolation tenant according to the adjusted resource quota.

[0011] Furthermore, the calculation determination unit is used to calculate the resource usage evaluation value according to the CPU usage rate and the memory usage rate, including:

[0012] The calculation and determination unit is used to respectively calculate the product of the CPU usage rate and the preset CPU weight, the product of the memory usage rate and the preset memory weight, and calculate the sum of the two products to obtain the resource usage evaluation value.

[0013] Furthermore, the calculation and determination unit makes a determination based on the resource usage evaluation value and a preset standard evaluation value to form a first determination result including:

[0014] The calculation determination unit determines that the preset resource quota of each named isolation tenant needs to be adjusted when the resource usage evaluation value is less than or equal to the preset standard evaluation value;

[0015] The calculation and determination unit determines that it is necessary to predict the change of resource usage trend when the resource usage evaluation value is greater than a preset standard evaluation value.

[0016] Furthermore, when the calculation determination unit determines that the preset resource quota of each named isolation tenant needs to be adjusted, the adjustment unit adjusts the preset resource quota according to the resource usage evaluation value, the preset standard evaluation value and the preset resource coefficient.

[0017] Further, the trend prediction module is used to determine the change trend of all resource usage evaluation values ​​within the preset prediction duration according to the first determination result, all resource usage evaluation values ​​within the preset prediction duration, and the preset standard resource usage change rate, including:

[0018] When the calculation and determination unit determines that a prediction of a resource usage trend change is required, the trend prediction module calculates the difference between the resource usage evaluation value at each time point within a preset prediction period and the resource usage evaluation value at the previous time point, and calculates the ratio of the difference to the resource usage evaluation value at the previous time point to obtain the resource usage change rate, accumulates the change rates at each time point to form a cumulative change rate, and when the cumulative change rate is greater than the preset standard resource usage change rate, determines that the change trend of all resource usage evaluation values ​​within the preset prediction period is an increasing trend.

[0019] Furthermore, the trend prediction module makes a determination based on the change trend, the total resource usage evaluation values ​​within the preset prediction time, and the preset evaluation value standard deviation threshold, including:

[0020] When the trend prediction module determines that the changing trend of all resource usage evaluation values ​​within the preset prediction period is an increasing trend, it calculates the standard deviation of all resource usage evaluation values ​​within the preset prediction period to obtain the evaluation value standard deviation. When the evaluation value standard deviation is greater than the preset evaluation value standard deviation threshold, the evaluation adjustment module determines that priority and correlation determination is required.

[0021] Furthermore, when the trend prediction module determines that priority and correlation determination is needed, the evaluation and adjustment module obtains the level value of the service level corresponding to each named isolation tenant in the preset service level agreement, and when the level value is greater than the preset standard level value, determines that the priority of the named isolation tenant is high.

[0022] Furthermore, the evaluation and adjustment module calculates the synchronization degree according to the priority, the total CPU usage rate within the preset prediction time, and the total memory usage rate, including:

[0023] When the evaluation and adjustment module determines that the priority of the named isolated tenant is high, it standardizes all CPU usage and all memory usage within the preset prediction duration to obtain standardized CPU usage and standardized memory usage, calculates the correlation coefficient of the standardized CPU usage and the standardized memory usage using the Pearson correlation coefficient calculation formula, and takes the absolute value of the correlation coefficient to obtain the synchronization degree.

[0024] Furthermore, the evaluation and adjustment module adjusts the preset resource quota of each named isolation tenant according to the synchronization degree and the preset standard synchronization degree range, including:

[0025] When the synchronization degree is less than the minimum value of the preset standard synchronization degree range, the evaluation and adjustment module determines that there is a waste of resources, and reduces the preset resource quota according to the synchronization degree, the minimum value of the preset standard synchronization degree range and the preset quota synchronization degree coefficient;

[0026] When the synchronization degree is greater than the maximum value of the preset standard synchronization degree range, the evaluation and adjustment module determines that the system performance has declined, and increases the preset resource quota according to the synchronization degree, the minimum value of the preset standard synchronization degree range and the preset quota synchronization degree coefficient.

[0027] A method for isolating and managing model training resources in a multi-tenant environment, based on the above-mentioned model training resource isolation and management system in a multi-tenant environment, includes:

[0028] Real-time monitoring of the CPU usage and memory usage of each named isolated tenant during model training;

[0029] Calculate the resource usage evaluation value according to the CPU usage rate and the memory usage rate, and make a judgment according to the resource usage evaluation value and the preset standard evaluation value to form a first judgment result, and adjust the preset resource quota of each named isolation tenant according to the first judgment result to form an adjusted resource quota;

[0030] Determine the change trend of the total resource usage evaluation value within the preset prediction duration according to the first determination result, the total resource usage evaluation value within the preset prediction duration, and the preset standard resource usage change rate, and make a determination according to the change trend, the total resource usage evaluation value within the preset prediction duration, and the preset evaluation value standard deviation threshold to form a second determination result;

[0031] Determine the priority of each named isolated tenant according to the second determination result, the service level of each named isolated tenant and the preset service level agreement, calculate the synchronization degree according to the priority, the total CPU usage rate and the total memory usage rate within the preset predicted time, and adjust the preset resource quota of each named isolated tenant according to the synchronization degree and the preset standard synchronization degree range to form an adjusted resource quota;

[0032] Resource allocation is performed for each named isolated tenant based on adjusting resource quotas.

[0033] Compared with the prior art, the beneficial effect of the present invention lies in that, by real-time monitoring of the resource usage of each named isolated tenant during the model training process, combined with preset evaluation criteria and trend predictions, the resource quota is dynamically adjusted to ensure the reasonable allocation and efficient utilization of resources. Through comprehensive evaluation of priority and synchronization, the system can timely correct the resource allocation strategy when resource usage fluctuates, avoid resource waste or excessive occupation, and improve the resource management efficiency of the overall system. This dynamic allocation mechanism not only improves resource utilization, but also adapts to changes in the needs of different tenants, ensuring the stability and flexibility of the system, and effectively solving the problems of inefficient resource utilization and unstable system performance due to reliance on static configuration.

[0034] Furthermore, by calculating the resource usage evaluation value by combining the weighted product of the CPU and memory usage, the overall resource usage of the system can be more accurately reflected.

[0035] Furthermore, by making judgments based on the comparison between resource usage evaluation values ​​and standard evaluation values, the system can dynamically adjust resource quotas to ensure the rational allocation of resources. At the same time, when resource usage exceeds the preset standard, it can predict trend changes in advance to avoid resource shortages or over-allocation.

[0036] Furthermore, through the dynamic adjustment mechanism, the system can timely optimize resource quota allocation when resource usage changes, ensuring the stability and efficiency of the model training process.

[0037] Furthermore, by adopting a trend prediction method based on the cumulative rate of change, the system can accurately capture the growth trend of resource usage, make resource adjustments in advance, and avoid resource waste or system overload.

[0038] Furthermore, by calculating the standard deviation of the evaluation value and comparing it with the preset threshold, the system can identify the volatility and uncertainty in resource use. When the volatility is large, the system can automatically trigger the priority and relevance determination, thereby optimizing resource allocation and ensuring that the resource requirements of key tasks are met first.

[0039] Furthermore, by comparing the service level value of the named isolated tenant with the standard level value, the system can effectively distinguish the advanced tenants who need priority resource protection, and ensure that mission-critical and high-priority tenants are given priority in resource allocation when resources are tight.

[0040] Furthermore, through normalization processing and Pearson correlation coefficient calculation, the evaluation of synchronization provides a quantitative way to measure the relationship between CPU and memory usage, effectively revealing the coordination of resource usage.

[0041] Furthermore, through dynamic adjustment based on synchronization, the system can respond to imbalances in resource usage and performance issues in real time, thereby optimizing the configuration of resource quotas.

[0042] Furthermore, by comprehensively considering resource usage evaluation values, trend predictions, priorities, and synchronization, the efficiency of resource usage is optimized. Real-time adjustment of resource quotas can effectively avoid resource waste and system performance degradation, and improve system stability and operating efficiency. At the same time, according to the tenant's service level and agreement, it ensures that the resource needs of different tenants are reasonably met. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the model training resource isolation and management system in a multi-tenant environment of this embodiment;

[0044] Figure 2 A determination logic diagram for forming a first determination result by the calculation determination unit of this embodiment;

[0045] Figure 3 A decision logic diagram for determining a change trend by the trend prediction module of this embodiment;

[0046] Figure 4 This is a flowchart of the model training resource isolation and management method in a multi-tenant environment of this embodiment. DETAILED DESCRIPTION

[0047] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0049] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0050] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] See also Figure 1 As shown, it is a schematic diagram of the model training resource isolation and management system in a multi-tenant environment of this embodiment;

[0052] A model training resource isolation and management system and method in a multi-tenant environment, comprising:

[0053] The data collection module is used to monitor the CPU usage and memory usage of each named isolated tenant during the model training process in real time;

[0054] A resource adjustment module, connected to the data acquisition module, comprising a calculation and determination unit and an adjustment unit, wherein the calculation and determination is used to calculate a resource usage evaluation value according to a CPU usage rate and a memory usage rate, and to make a determination according to the resource usage evaluation value and a preset standard evaluation value to form a first determination result, and the adjustment unit is used to adjust a preset resource quota of each named isolation tenant according to the first determination result to form an adjusted resource quota;

[0055] a trend prediction module, connected to the data acquisition module and the resource adjustment module respectively, for determining the change trend of the total resource usage evaluation value within the preset prediction time according to the first judgment result, the total resource usage evaluation value within the preset prediction time and the preset standard resource usage change rate, and making a judgment according to the change trend, the total resource usage evaluation value within the preset prediction time and the preset evaluation value standard deviation threshold to form a second judgment result;

[0056] An evaluation and adjustment module, connected to the data collection module and the trend prediction module respectively, for determining the priority of each named isolation tenant according to the second determination result, the service level of each named isolation tenant and the preset service level agreement, calculating the synchronization degree according to the priority, the total CPU usage rate and the total memory usage rate within the preset prediction time, and adjusting the preset resource quota of each named isolation tenant according to the synchronization degree and the preset standard synchronization degree range to form an adjusted resource quota;

[0057] The execution module is connected to the resource adjustment module and the evaluation adjustment module respectively, and is used to perform resource allocation for each named isolation tenant according to the adjusted resource quota.

[0058] Named isolation tenants refer to instances or virtual environments in which different tenants (such as enterprises, users or applications) run independently through naming isolation technology in a multi-tenant cloud computing environment. Each tenant has its own dedicated computing resources, storage space and network configuration, but these resources are isolated and allocated on the same physical infrastructure to ensure that data and resources between different tenants do not interfere with each other.

[0059] The preset standard evaluation value is a benchmark value for measuring resource usage, which depends on historical data and resource demand models and is usually set to 80% of the average system usage. In this embodiment, it is set to 85%, which helps to ensure the appropriateness of resource allocation.

[0060] The preset resource quota is the initial amount of resources allocated to each tenant, which depends on the tenant's needs and historical usage. It is usually set to 90% of the tenant's peak demand. In this embodiment, it is set to 95%, which helps to cope with sudden changes in resource demand.

[0061] The preset prediction duration is the time window for predicting resource usage trends, which depends on the duration of model training and the system response speed. It is usually set to 15 minutes. In this embodiment, it is set to 10 minutes, which helps to respond to changes in resource demand more quickly.

[0062] The preset standard resource usage change rate is a benchmark rate used to determine the resource usage change trend, which depends on the volatility of system resource usage and is usually set to 10%. In this embodiment, it is set to 8%, which helps to capture the resource usage change trend more accurately.

[0063] The preset evaluation value standard deviation threshold is used to determine whether the resource usage fluctuation exceeds the normal range. It depends on the discreteness of historical data and is usually set to 2%. In this embodiment, it is set to 1.5%, which helps to improve the sensitivity and accuracy of the system.

[0064] The service level of each named isolated tenant indicates the importance and priority of each tenant, which depends on the tenant's contract and service requirements and is usually determined by negotiation between the tenant and the service provider. In this embodiment, it is set according to the contract, which helps to optimize resource allocation and meet the needs of key tenants.

[0065] The preset service level agreement defines the relationship between service level and resource allocation, which depends on the content of the service agreement and the needs of the tenants. It is usually set as clear service terms. In this embodiment, it is set according to the requirements of the tenants, which helps to ensure the service quality and the reasonable allocation of resources.

[0066] The preset standard synchronization range is used to evaluate the consistency of CPU and memory usage, and depends on the balance requirements of the system. It is usually set to a range of ±5%. In this embodiment, it is set to ±3%, which helps to ensure efficient use of resources and system stability.

[0067] The data acquisition module monitors the CPU usage and memory usage of each named isolated tenant in the model training process in real time, and passes the data to the resource adjustment module. The resource adjustment module adjusts the resource quota of each tenant according to the judgment result of the calculation judgment unit. The trend prediction module predicts the future resource usage trend based on the first judgment result and the preset resource usage change rate, and further optimizes the resource allocation according to the tenant's priority and synchronization through the evaluation adjustment module. Finally, the execution module allocates resources to each tenant according to the adjusted resource quota.

[0068] By real-time monitoring of resource usage during model training of each named isolated tenant, combined with preset evaluation criteria and trend predictions, resource quotas are dynamically adjusted to ensure reasonable allocation and efficient utilization of resources. Through comprehensive evaluation of priority and synchronization, the system can promptly correct resource allocation strategies when resource usage fluctuates, avoid resource waste or excessive occupation, and improve the resource management efficiency of the overall system. This dynamic allocation mechanism not only improves resource utilization, but also adapts to changing needs of different tenants, ensuring the stability and flexibility of the system.

[0069] Specifically, the calculation determination unit is used to calculate the resource usage evaluation value according to the CPU usage rate and the memory usage rate, including:

[0070] The calculation and determination unit is used to respectively calculate the product of the CPU usage rate and the preset CPU weight, the product of the memory usage rate and the preset memory weight, and calculate the sum of the two products to obtain the resource usage evaluation value.

[0071] The calculation and determination unit first calculates the product of the CPU usage rate and the preset CPU weight, and the product of the memory usage rate and the preset memory weight, respectively. Then, the two products are added together to obtain a resource usage evaluation value, which is used to evaluate the current resource usage and provide a basis for subsequent resource adjustment and allocation.

[0072] By calculating the resource usage evaluation value by combining the weighted product of the CPU and memory usage, the overall resource usage of the system can be more accurately reflected.

[0073] The preset CPU weight is a parameter used to calculate the resource usage evaluation value, which depends on the system's assessment of the importance of the CPU resources and is usually set between 0.5 and 0.8. In this embodiment, it is set to 0.6. This setting can balance the influence of the CPU in the overall resource usage and ensure that computing-intensive tasks receive sufficient resource support.

[0074] The preset memory weight is another parameter used to calculate the resource usage evaluation value, which depends on the system's assessment of the importance of memory resources and is usually set between 0.2 and 0.5. In this embodiment, it is set to 0.4. This setting helps to provide sufficient resource allocation in memory-intensive tasks and avoid performance problems caused by insufficient memory.

[0075] Please continue reading Figure 2 As shown, it is a determination logic diagram of the calculation determination unit in this embodiment to form a first determination result;

[0076] Specifically, the calculation and determination unit makes a determination based on the resource usage evaluation value and the preset standard evaluation value to form a first determination result including:

[0077] The calculation determination unit determines that the preset resource quota of each named isolation tenant needs to be adjusted when the resource usage evaluation value is less than or equal to the preset standard evaluation value;

[0078] The calculation and determination unit determines that it is necessary to predict the change of resource usage trend when the resource usage evaluation value is greater than a preset standard evaluation value.

[0079] The calculation and judgment unit first compares the resource usage evaluation value with the preset standard evaluation value. When the resource usage evaluation value is less than or equal to the preset standard evaluation value, it is determined that the current resource allocation of the system is appropriate and the preset resource quota of each named isolated tenant needs to be adjusted; when the resource usage evaluation value is greater than the preset standard evaluation value, the system determines that there may be a risk of insufficient resources and it is necessary to initiate a prediction of changes in resource usage trends in order to identify and respond to potential fluctuations in resource demand in advance.

[0080] By comparing the resource usage evaluation value with the standard evaluation value, the system can dynamically adjust the resource quota to ensure the rational allocation of resources. At the same time, when resource usage exceeds the preset standard, it can predict trend changes in advance to avoid resource shortages or over-allocation.

[0081] Specifically, when the calculation and determination unit determines that the preset resource quota of each named isolated tenant needs to be adjusted, the adjustment unit adjusts the preset resource quota according to the resource usage evaluation value, the preset standard evaluation value and the preset resource coefficient, wherein Q'=Q+(V-V0)×C, Q' is the adjusted resource quota, Q is the preset resource quota, V is the resource usage evaluation value, V0 is the preset standard evaluation value, and C is the preset resource coefficient.

[0082] The preset resource coefficient is a parameter used to adjust the range of resource quota changes. It determines how the resource quota should be adjusted when there is a difference between the resource usage evaluation value and the preset standard evaluation value. This coefficient usually depends on the elasticity requirements of the system, the priority of the tenant, and the overall availability of resources. It is usually set according to the system's requirements for resource sensitivity, and may be a smaller value such as 0.1 or 0.2 to ensure a smooth adjustment range. In this embodiment, the preset resource coefficient is set to 0.15. The purpose is to make appropriate resource quota adjustments when there is a difference between the resource usage evaluation value and the standard evaluation value, to ensure that when the resource usage demand fluctuates, the system can quickly and without excessively adjust resource allocation.

[0083] When the calculation and determination unit determines that the resource usage evaluation value needs to adjust the preset resource quota, the adjustment unit dynamically adjusts the preset resource quota of each named isolated tenant based on the difference between the resource usage evaluation value and the preset standard evaluation value and the preset resource coefficient to ensure that resource allocation is more in line with actual needs.

[0084] Through the dynamic adjustment mechanism, the system can optimize resource quota allocation in a timely manner when resource usage changes, ensuring the stability and efficiency of the model training process.

[0085] Please continue reading Figure 3 As shown, it is a decision logic diagram of the trend prediction module of this embodiment for determining the change trend;

[0086] Specifically, the trend prediction module is used to determine the change trend of all resource usage evaluation values ​​within the preset prediction duration according to the first determination result, all resource usage evaluation values ​​within the preset prediction duration, and the preset standard resource usage change rate, including:

[0087] When the calculation and determination unit determines that a prediction of a resource usage trend change is required, the trend prediction module calculates the difference between the resource usage evaluation value at each time point within a preset prediction period and the resource usage evaluation value at the previous time point, and calculates the ratio of the difference to the resource usage evaluation value at the previous time point to obtain the resource usage change rate, accumulates the change rates at each time point to form a cumulative change rate, and when the cumulative change rate is greater than the preset standard resource usage change rate, determines that the change trend of all resource usage evaluation values ​​within the preset prediction period is an increasing trend.

[0088] The trend prediction module monitors resource usage during model training, calculates the difference between the resource usage evaluation value at each time point within the preset prediction duration and the previous time point, and calculates the resource usage change rate through the ratio. The change rates at each time point are accumulated to form a cumulative change rate. If the cumulative change rate exceeds the preset standard resource usage change rate, the system determines that the overall trend of resource usage is an increasing trend. This process provides a prediction basis for resource management, enabling the system to adjust resource quotas in advance to avoid resource shortages.

[0089] By adopting a trend prediction method based on the cumulative rate of change, the system can accurately capture the growth trend of resource usage, make resource adjustments in advance, and avoid resource waste or system overload.

[0090] Specifically, the trend prediction module makes a determination based on the change trend, the total resource usage evaluation value within the preset prediction time, and the preset evaluation value standard deviation threshold, including:

[0091] When the trend prediction module determines that the changing trend of all resource usage evaluation values ​​within the preset prediction period is an increasing trend, it calculates the standard deviation of all resource usage evaluation values ​​within the preset prediction period to obtain the evaluation value standard deviation. When the evaluation value standard deviation is greater than the preset evaluation value standard deviation threshold, the evaluation adjustment module determines that priority and correlation determination is required.

[0092] After determining that the change trend of the resource usage evaluation value is an increasing trend, the trend prediction module further calculates the standard deviation of all resource usage evaluation values ​​within the preset prediction period to obtain the evaluation value standard deviation. If the evaluation value standard deviation is greater than the preset evaluation value standard deviation threshold, the evaluation adjustment module will further determine whether priority and relevance determination is required to ensure the rationality and effectiveness of resource allocation.

[0093] By calculating the standard deviation of the evaluation value and comparing it with the preset threshold, the system can identify the volatility and uncertainty in resource use. When the volatility is large, the system can automatically trigger the priority and relevance determination, thereby optimizing resource allocation and ensuring that the resource requirements of key tasks are met first.

[0094] Specifically, when the trend prediction module determines that priority and correlation determination is needed, the evaluation and adjustment module obtains the level value of the service level corresponding to each named isolation tenant in the preset service level agreement. When the level value is greater than the preset standard level value, the priority of the named isolation tenant is determined to be high.

[0095] When the trend prediction module determines that priority and relevance determination is required, the evaluation and adjustment module obtains the level value of the service level corresponding to each named isolation tenant in the preset service level agreement and compares it with the preset standard level value. If the level value of a named isolation tenant is greater than the preset standard level value, the priority of the tenant is determined to be high, thereby providing a basis for subsequent resource allocation.

[0096] The preset standard level value is a benchmark value used to determine the service priority of each named isolation tenant. Depending on the service agreement requirements and business importance, the standard level value can be set according to specific business needs and system design, usually based on industry standards or requirements in the service agreement. In specific implementations, the standard level value can be set to a specific value based on the terms of the service agreement and business needs. For example, the value range is between 1 and 10. The higher the value, the higher the priority, which helps the system clearly distinguish the service priorities of different tenants.

[0097] By comparing the service level value of the named isolated tenant with the standard level value, the system can effectively distinguish the advanced tenants who need priority resource protection, and ensure that mission-critical and high-priority tenants are given priority in resource allocation when resources are tight.

[0098] Specifically, the evaluation and adjustment module calculates the synchronization degree according to the priority, the total CPU usage rate within the preset prediction time, and the total memory usage rate, including:

[0099] When the evaluation and adjustment module determines that the priority of the named isolated tenant is high, it standardizes all CPU usage and all memory usage within the preset prediction duration to obtain standardized CPU usage and standardized memory usage, calculates the correlation coefficient of the standardized CPU usage and the standardized memory usage using the Pearson correlation coefficient calculation formula, and takes the absolute value of the correlation coefficient to obtain the synchronization degree.

[0100] After the evaluation and adjustment module determines that the tenant's priority is high, the module first standardizes the CPU usage and memory usage within the preset forecast duration to ensure the comparability of the data. Then, the Pearson correlation coefficient calculation formula is used to perform a correlation analysis on the standardized CPU usage and memory usage. Finally, the absolute value of the calculated correlation coefficient is taken as the synchronization degree. This process ensures the uniformity of different indicators and accurately reflects the degree of synchronization of resource usage.

[0101] Through normalization processing and Pearson correlation coefficient calculation, the evaluation of synchronization provides a quantitative way to measure the relationship between CPU and memory usage, effectively revealing the coordination of resource usage.

[0102] Specifically, the evaluation and adjustment module adjusts the preset resource quota of each named isolated tenant according to the synchronization degree and the preset standard synchronization degree range, including:

[0103] When the synchronization degree is less than the minimum value of the preset standard synchronization degree range, the evaluation and adjustment module determines that there is a waste of resources, and reduces the preset resource quota according to the synchronization degree, the minimum value of the preset standard synchronization degree range and the preset quota synchronization degree coefficient;

[0104] When the synchronization degree is greater than the maximum value of the preset standard synchronization degree range, the evaluation and adjustment module determines that the system performance has declined, and increases the preset resource quota according to the synchronization degree, the minimum value of the preset standard synchronization degree range and the preset quota synchronization degree coefficient.

[0105] When the synchronization degree is less than the minimum value of the standard synchronization degree range:

[0106] Q'=QK×(Smin-S), Q' is the adjusted resource quota, Q is the preset resource quota, K is the quota synchronization coefficient, Smin is the minimum value of the preset standard synchronization range, and S is the synchronization degree.

[0107] The preset quota synchronization coefficient is a parameter used to adjust the resource quota to cope with the deviation between the synchronization and the preset standard synchronization range. The setting of the quota synchronization coefficient depends on the resource requirements, usage patterns and business priorities of the system. It determines the extent to which the resource quota is adjusted when the synchronization exceeds or falls below the standard synchronization range. It is usually adjusted according to actual business needs and system performance requirements. Its setting range can be a value less than 1 to avoid over-adjustment, or greater than 1 to respond to synchronization changes more quickly. This embodiment is set to 0.1 (when decreasing) and 1.5 (when increasing). This means that when the synchronization is lower than the standard range, the quota will be reduced by 10%, and when the synchronization is higher than the standard range, the quota will be increased by 50%, which can effectively adjust the resource quota when the synchronization deviates.

[0108] The evaluation and adjustment module determines resource usage by measuring the degree of synchronization. When the degree of synchronization is less than the minimum value of the preset standard synchronization range, it indicates that there is a waste of resources. At this time, the module reduces the preset resource quota according to the degree of synchronization, the minimum value of the standard synchronization range, and the quota synchronization coefficient; and when the degree of synchronization is greater than the maximum value of the standard synchronization range, it indicates that the system performance has declined. The module increases the preset resource quota according to the degree of synchronization, the maximum value of the standard synchronization range, and the quota synchronization coefficient. This adjustment process helps to dynamically optimize resource allocation, avoid waste, and improve system performance.

[0109] Through dynamic adjustments based on synchronization, the system can respond to resource usage imbalances and performance issues in real time, thereby optimizing the configuration of resource quotas.

[0110] Please continue reading Figure 4 As shown, it is a flow chart of the model training resource isolation and management method in a multi-tenant environment of this embodiment;

[0111] A method for isolating and managing model training resources in a multi-tenant environment, based on the above-mentioned model training resource isolation and management system in a multi-tenant environment, includes:

[0112] Real-time monitoring of the CPU usage and memory usage of each named isolated tenant during model training;

[0113] Calculate the resource usage evaluation value according to the CPU usage rate and the memory usage rate, and make a judgment according to the resource usage evaluation value and the preset standard evaluation value to form a first judgment result, and adjust the preset resource quota of each named isolation tenant according to the first judgment result to form an adjusted resource quota;

[0114] Determine the change trend of the total resource usage evaluation value within the preset prediction duration according to the first determination result, the total resource usage evaluation value within the preset prediction duration, and the preset standard resource usage change rate, and make a determination according to the change trend, the total resource usage evaluation value within the preset prediction duration, and the preset evaluation value standard deviation threshold to form a second determination result;

[0115] Determine the priority of each named isolated tenant according to the second determination result, the service level of each named isolated tenant and the preset service level agreement, calculate the synchronization degree according to the priority, the total CPU usage rate and the total memory usage rate within the preset predicted time, and adjust the preset resource quota of each named isolated tenant according to the synchronization degree and the preset standard synchronization degree range to form an adjusted resource quota;

[0116] Resource allocation is performed for each named isolated tenant based on adjusting resource quotas.

[0117] First, the CPU and memory resources used by each named isolated tenant during the model training process are monitored in real time. The resource usage evaluation values ​​of the CPU usage rate and the memory usage rate are calculated and compared with the preset standard evaluation values ​​to form a first judgment result, and then the resource quota is adjusted accordingly. Next, based on the first judgment result, the resource usage evaluation value within the predicted duration, and the standard resource usage change rate, the change trend of resource usage is determined and a second judgment result is formed. Then, the tenant priority is determined by analyzing the second judgment result, the tenant service level and the service level agreement, and the synchronization is calculated to adjust the resource quota. Finally, the execution module allocates resources according to the adjusted quota to ensure the optimization of resource usage and the improvement of system performance.

[0118] By comprehensively considering resource usage evaluation values, trend predictions, priorities, and synchronization, the efficiency of resource usage is optimized. Real-time adjustment of resource quotas can effectively avoid resource waste and system performance degradation, and improve system stability and operating efficiency. At the same time, according to the tenant's service level and agreement, it ensures that the resource needs of different tenants are reasonably met.

[0119] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A model training resource isolation and management system in a multi-tenant environment, characterized in that: include: The data collection module is used to monitor the CPU usage and memory usage of each named isolated tenant during the model training process in real time; A resource adjustment module, connected to the data acquisition module, comprising a calculation and determination unit and an adjustment unit, wherein the calculation and determination is used to calculate a resource usage evaluation value according to a CPU usage rate and a memory usage rate, and to make a determination according to the resource usage evaluation value and a preset standard evaluation value to form a first determination result, and the adjustment unit is used to adjust a preset resource quota of each named isolation tenant according to the first determination result to form an adjusted resource quota; a trend prediction module, connected to the data acquisition module and the resource adjustment module respectively, for determining the change trend of the total resource usage evaluation value within the preset prediction time according to the first judgment result, the total resource usage evaluation value within the preset prediction time and the preset standard resource usage change rate, and making a judgment according to the change trend, the total resource usage evaluation value within the preset prediction time and the preset evaluation value standard deviation threshold to form a second judgment result; An evaluation and adjustment module, connected to the data collection module and the trend prediction module respectively, for determining the priority of each named isolation tenant according to the second determination result, the service level of each named isolation tenant and the preset service level agreement, calculating the synchronization degree according to the priority, the total CPU usage rate and the total memory usage rate within the preset prediction time, and adjusting the preset resource quota of each named isolation tenant according to the synchronization degree and the preset standard synchronization degree range to form an adjusted resource quota; an execution module, connected to the resource adjustment module and the evaluation adjustment module respectively, for executing resource allocation for each named isolation tenant according to the adjusted resource quota; The evaluation adjustment module adjusts the preset resource quota of each named isolated tenant according to the synchronization degree and the preset standard synchronization degree range, including: When the synchronization degree is less than the minimum value of the preset standard synchronization degree range, the evaluation and adjustment module determines that there is a waste of resources, and reduces the preset resource quota according to the synchronization degree, the minimum value of the preset standard synchronization degree range and the preset quota synchronization degree coefficient; When the synchronization degree is greater than the maximum value of the preset standard synchronization degree range, the evaluation and adjustment module determines that the system performance has declined, and increases the preset resource quota according to the synchronization degree, the minimum value of the preset standard synchronization degree range and the preset quota synchronization degree coefficient.

2. The model training resource isolation and management system in a multi-tenant environment according to claim 1, characterized in that: The calculation and determination unit is used to calculate the resource usage evaluation value according to the CPU usage rate and the memory usage rate, including: The calculation and determination unit is used to respectively calculate the product of the CPU usage rate and the preset CPU weight, the product of the memory usage rate and the preset memory weight, and calculate the sum of the two products to obtain the resource usage evaluation value.

3. The model training resource isolation and management system in a multi-tenant environment according to claim 2, characterized in that: The calculation and determination unit makes a determination based on the resource usage evaluation value and the preset standard evaluation value to form a first determination result including: The calculation determination unit determines that the preset resource quota of each named isolation tenant needs to be adjusted when the resource usage evaluation value is less than or equal to the preset standard evaluation value; The calculation and determination unit determines that it is necessary to predict the change of resource usage trend when the resource usage evaluation value is greater than a preset standard evaluation value.

4. The model training resource isolation and management system in a multi-tenant environment according to claim 3, characterized in that: When the calculation determination unit determines that the preset resource quota of each named isolation tenant needs to be adjusted, the adjustment unit adjusts the preset resource quota according to the resource usage evaluation value, the preset standard evaluation value and the preset resource coefficient.

5. The model training resource isolation and management system in a multi-tenant environment according to claim 4, characterized in that: The trend prediction module is used to determine the change trend of all resource usage evaluation values ​​within the preset prediction duration according to the first determination result, all resource usage evaluation values ​​within the preset prediction duration, and the preset standard resource usage change rate, including: When the calculation and determination unit determines that a prediction of a resource usage trend change is required, the trend prediction module calculates the difference between the resource usage evaluation value at each time point within a preset prediction period and the resource usage evaluation value at the previous time point, and calculates the ratio of the difference to the resource usage evaluation value at the previous time point to obtain the resource usage change rate, accumulates the change rates at each time point to form a cumulative change rate, and when the cumulative change rate is greater than the preset standard resource usage change rate, determines that the change trend of all resource usage evaluation values ​​within the preset prediction period is an increasing trend.

6. The model training resource isolation and management system in a multi-tenant environment according to claim 5, characterized in that: The trend prediction module makes a determination based on the change trend, the total resource usage evaluation value within the preset prediction time, and the preset evaluation value standard deviation threshold, including: When the trend prediction module determines that the changing trend of all resource usage evaluation values ​​within the preset prediction period is an increasing trend, it calculates the standard deviation of all resource usage evaluation values ​​within the preset prediction period to obtain the evaluation value standard deviation. When the evaluation value standard deviation is greater than the preset evaluation value standard deviation threshold, the evaluation adjustment module determines that priority and correlation determination is required.

7. The model training resource isolation and management system in a multi-tenant environment according to claim 6, characterized in that: When the trend prediction module determines that priority and correlation determination is needed, the evaluation and adjustment module obtains the level value of the service level corresponding to each named isolation tenant in the preset service level agreement. When the level value is greater than the preset standard level value, the priority of the named isolation tenant is determined to be high.

8. The model training resource isolation and management system in a multi-tenant environment according to claim 7, characterized in that: The evaluation and adjustment module calculates the synchronization degree according to the priority, the total CPU usage rate within the preset prediction time, and the total memory usage rate, including: When the evaluation and adjustment module determines that the priority of the named isolated tenant is high, it standardizes all CPU usage and all memory usage within the preset prediction duration to obtain standardized CPU usage and standardized memory usage, calculates the correlation coefficient of the standardized CPU usage and the standardized memory usage using the Pearson correlation coefficient calculation formula, and takes the absolute value of the correlation coefficient to obtain the synchronization degree.

9. A method for isolating and managing model training resources in a multi-tenant environment, based on the model training resource isolation and management system in a multi-tenant environment according to any one of claims 1 to 8, characterized in that: include: Real-time monitoring of the CPU usage and memory usage of each named isolated tenant during model training; Calculate the resource usage evaluation value according to the CPU usage rate and the memory usage rate, and make a judgment according to the resource usage evaluation value and the preset standard evaluation value to form a first judgment result, and adjust the preset resource quota of each named isolation tenant according to the first judgment result to form an adjusted resource quota; Determine the change trend of the total resource usage evaluation value within the preset prediction duration according to the first determination result, the total resource usage evaluation value within the preset prediction duration, and the preset standard resource usage change rate, and make a determination according to the change trend, the total resource usage evaluation value within the preset prediction duration, and the preset evaluation value standard deviation threshold to form a second determination result; Determine the priority of each named isolated tenant according to the second determination result, the service level of each named isolated tenant and the preset service level agreement, calculate the synchronization degree according to the priority, the total CPU usage rate and the total memory usage rate within the preset predicted time, and adjust the preset resource quota of each named isolated tenant according to the synchronization degree and the preset standard synchronization degree range to form an adjusted resource quota; Allocate resources to each named isolated tenant based on the adjusted resource quota; Adjust the preset resource quotas of each named isolation tenant based on the synchronization degree and the preset standard synchronization range, including: When the synchronization degree is less than the minimum value of the preset standard synchronization degree range, it is determined that there is a waste of resources, and the preset resource quota is reduced according to the synchronization degree, the minimum value of the preset standard synchronization degree range, and the preset quota synchronization degree coefficient; When the synchronization degree is greater than the maximum value of the preset standard synchronization degree range, it is determined that the system performance is degraded, and the preset resource quota is increased according to the synchronization degree, the minimum value of the preset standard synchronization degree range and the preset quota synchronization degree coefficient.

Images