Resource adjustment method, device and equipment of system

Abstract

The invention discloses a resource adjustment method, device and equipment of a system. Firstly, the current state and the next action of a target system can be obtained, the next state is obtained after the action is executed, and default penalty and a multi-dimensional reward signal matrix in the process that the current state is changed to the next state are calculated. And performing nonlinear combination on the reward signal matrix to form a preliminary reward function, and calculating a dynamic discount factor of a next state at the same time. And constructing a total reward function in combination with the default penalty and the preliminary reward function, and iteratively updating the value function based on the reward function and the dynamic discount factor until convergence. And finally, carrying out effective resource adjustment on the target system according to the converged value function. According to the method, flexible default punishment is realized by quantifying the constraint violation degree, the expression ability of multi-dimensional rewards is improved by adopting a nonlinear reward combination, and the dynamic discount factor is introduced to consider short-term and long-term benefits, so that the adaptability, the optimization effect and the continuous performance guarantee of a resource scheduling algorithm are remarkably enhanced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and particularly to a method, device, and equipment for resource adjustment of a system. Background Art

[0002] In the field of computing resource scheduling, reinforcement learning methods have gradually become a research hotspot due to their adaptability and powerful optimization capabilities. However, current technologies mostly use heuristic rules or relatively simple reinforcement learning strategies for resource allocation, and these methods have obvious deficiencies when dealing with complex multi-dimensional optimization goals. Especially in the design of the reward function, there are three major problems in the reward function design: First, multi-dimensional rewards usually adopt a simple linear combination with fixed weights, which is difficult to reflect the complex relationships and performance trade-offs between indicators; second, hard constraints are processed through threshold limits, resulting in a limited feasible solution space and a lack of quantification of the degree of constraint violation, affecting the algorithm's exploration ability and the quality of the solution; third, too much attention is paid to short-term immediate benefits, ignoring long-term decisions and system dynamics, making it difficult to achieve global optimization and continuous performance guarantee. Therefore, resource scheduling combined with the current reward function has significant defects such as fixed weights, rigid constraint handling, and lack of a long-term perspective, and it is difficult to meet the optimization requirements of complex systems.

[0003] Therefore, how to design a reward function that takes into account multi-dimensional dynamic weights, adaptive constraint violation penalties, and long-term benefit optimization to improve the decision-making effect of reinforcement learning in complex resource scheduling and the overall performance of the system is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0004] Based on the above problems, the present application provides a method, device, and equipment for resource adjustment of a system, which can provide a reward function that takes into account multi-dimensional dynamic weights, adaptive constraint violation penalties, and long-term benefit optimization to improve the decision-making effect of reinforcement learning in complex resource scheduling and the overall performance of the system.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] A method for resource adjustment of a system, the method comprising:

[0007] Obtain the current system state and the next system action of the target system; the current system state includes the resource utilization rate, service quality, system energy consumption, and system cost of the target system;

[0008] Execute the next system action to obtain the next system state, and calculate the default penalty during the change process from the current system state to the next system state calculation state in combination with the constraint condition set; the default penalty is obtained by quantifying and calculating based on the degree of constraint violation during the change process from the current system state to the next system state calculation state;

[0009] Combined with the reward dimension mechanism, calculate the reward signals for each reward dimension according to the current system state and the next system state to obtain a reward signal matrix;

[0010] Perform a non-linear reward combination on the reward signal matrix to obtain a preliminary reward function, and calculate the dynamic discount factor of the next system state;

[0011] Combine the default penalty and the preliminary reward function to calculate the total reward function, and perform dynamic iterative update calculation on the value function by combining the total reward function and the dynamic discount factor of the next system state until a convergent value function is obtained;

[0012] Adjust the resources of the target system by combining the convergent value function and the current system state.

[0013] In a possible implementation manner, the combining the default penalty and the preliminary reward function to calculate the total reward function, and performing dynamic iterative update calculation on the value function by combining the total reward function and the dynamic discount factor of the next system state until a convergent value function is obtained includes:

[0014] Use the dynamic iterative update calculation formula of the value function to calculate the value function by combining the total reward function and the dynamic discount factor of the next system state;

[0015] If the value function does not converge, iteratively adjust and calculate the dynamic parameters and hyperparameters in the formulas used for calculating the preliminary reward function, the dynamic discount factor, and the value function until the convergent value function is obtained and then stop.

[0016] In a possible implementation manner, the calculation formula for the degree of constraint violation is: v j = max(0, g j (s t , a t ))), where j = 1, 2,..., m, and m represents the total number of constraints in the constraint condition set; v j represents the degree of constraint violation of the j-th constraint condition in the constraint condition set; s t represents the current system state; a t represents the next system action; g j (s t , a t ) represents the constraint function value of the corresponding constraint condition, and g j (s t , a t ) is used to measure the value of the constraint function in state s tExecute action a below t After that, check whether the j-th constraint condition in the constraint condition set is satisfied, g j (s t , a t ) is less than or equal to 0;

[0017] The calculation formula of the default penalty is: , where p j (v j ) = -β j ·h(v j ); p j (v j ) represents the default penalty of v j ; h(v j ) represents the penalty shape function, h(v j ) includes h(v j ) = v j , h(v j ) = v j ², h(v j ) = -1, h(v j ) = log(1 + λv j ), where λ represents the shape hyperparameter; β j is a dynamic parameter representing the adaptive penalty coefficient, β j = β j t-1 ·(1 + δ·sign( -ε j ), δ represents the adjustment step size, β j t-1 represents the adaptive penalty coefficient during the previous iteration update, represents the average value of the constraint violation degree during the previous iteration update process; ε j represents the violation threshold.

[0018] In a possible implementation manner, the non-linear reward combination of the reward signal matrix to obtain the preliminary reward function includes:

[0019] Construct the correlation matrix of the reward signal matrix, and normalize each dimension reward in the reward signal matrix to obtain the normalized reward matrix;

[0020] Use the non-linear combination function to combine the correlation matrix, and combine each dimension reward in the normalized reward matrix to obtain the preliminary reward function;

[0021] Among them, the correlation matrix is: , = 1, Indicates that each reward dimension is completely positively correlated with itself; Indicates the degree of correlation between the i-th indicator and the j-th indicator, ∈[-1, 1]; = ;

[0022] The non-linear combination function is: , where R is the preliminary reward function; and Indicates the reward in the reward signal matrix; Indicates the cross-term function; w i is a dynamic parameter, representing the i adaptive weight of r; w i = w i （t-1） · (1 + η · sign(△Pi) · |△Pi| k ); w i （t -1） Indicates the weight during the previous iteration update ; η is a dynamic parameter, representing the learning rate; sign(△Pi) represents the sign function that determines whether to adjust and the adjustment direction according to △Pi; k is a dynamic parameter, representing the adjustment sensitivity parameter; △Pi represents the performance gap, △Pi = (Pi - Ti) / Ti, where Pi represents the current system state used when calculating the reward in the reward signal matrix ; Ti represents the target state of Pi.

[0023] In a possible implementation, the calculation formula for the dynamic discount factor is: γ(s) = γ base + △γ(s);

[0024] where γ(s) represents the dynamic discount factor of the system state; γ base is a dynamic parameter, representing the base discount factor; △γ(s) represents the discount adjustment term of the system state, △γ(s) = ω1 · φ1(load(s)) + ω2 · φ2(scarcity(s)) + ω3 · φ3(urgency(s)); load(s) represents the system load influencing factor, scarcity(s) represents the resource scarcity influencing factor; urgency(s) represents the task urgency influencing factor; φi(·) is a mapping function that maps each factor to a discount adjustment value; ωi is a dynamic parameter, representing the preset weight coefficient; the value of i is 1, 2, 3.

[0025] In a possible implementation, the calculation formula for the total reward function is: R total = R + ;

[0026] wherein, R is the preliminary reward function; is the default penalty, and m represents the total number of constraints in the constraint set.

[0027] In a possible implementation, the dynamic iterative update calculation formula of the value function is: Q = Q(s t , a t ) + α[R total + γ(s t+1 )·max α Q(s t+1 , α) - Q(s t , a t )];

[0028] wherein, s t represents the current system state; s t+1 represents the next system state obtained after executing the next system action a t ; Q(s t , a t ) represents the expected cumulative reward value of executing the next system action a t in the current system state s t ; α is a dynamic parameter representing the learning rate; R total represents the total reward function; γ(s t+1 ) represents the dynamic discount factor of the next system state; max α Q(s t+1 , α) represents the maximum value of all possible actions in the next system state s t+1 .

[0029] In a possible implementation, the method further includes:

[0030] Smoothing the adaptive weight w i to obtain a smoothed adaptive weight w i ´, and using the smoothed adaptive weight w i ´ to replace the adaptive weight w i to calculate the preliminary reward function R;

[0031] The smoothed adaptive weight w i ´ = (1 - ρ)·w i + ρ·w i （t-1） , where ρ represents a smoothing coefficient for controlling the weight smoothness.

[0032] A resource adjustment device for a system, the device includes:

[0033] An acquisition unit for acquiring the current system state and the next system action of the target system; the current system state includes the resource utilization rate, service quality, system energy consumption, and system cost of the target system;

[0034] An execution unit for executing the next system action to obtain the next system state;

[0035] A default penalty calculation unit for calculating the default penalty in the process of changing from the current system state to the calculation state of the next system state in combination with a set of constraint conditions; the default penalty is obtained through quantitative calculation based on the degree of constraint violation in the process of changing from the current system state to the calculation state of the next system state;

[0036] A reward signal matrix calculation unit for calculating the reward signals of each reward dimension according to the current system state and the next system state in combination with a reward dimension mechanism to obtain a reward signal matrix;

[0037] A preliminary reward function calculation unit for performing non - linear reward combination on the reward signal matrix to obtain a preliminary reward function;

[0038] A dynamic discount factor calculation unit for calculating the dynamic discount factor of the next system state;

[0039] A total reward function calculation unit for calculating the total reward function by combining the default penalty and the preliminary reward function to obtain the total reward function;

[0040] A value function calculation unit for calculating the total reward function by combining the default penalty and the preliminary reward function to obtain the total reward function, and dynamically iteratively updating the value function in combination with the total reward function and the dynamic discount factor of the next system state until a convergent value function is obtained;

[0041] A resource adjustment unit for adjusting the resources of the target system in combination with the convergent value function and the current system state.

[0042] A resource adjustment device for a system, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the computer program, implementing the resource adjustment method for the system according to any one of claims 1 - 8.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] The present application provides a method, an apparatus, and a device for resource adjustment of a system. Specifically, when executing the method for resource adjustment of the system provided in the embodiments of the present application, first, the current system state of the target system (including resource utilization rate, quality of service, system energy consumption, and system cost) and the next system action can be obtained. Execute this action to obtain the next system state, and calculate the default penalty during the change process from the current system state to the next system state in combination with the constraint set. Next, in combination with the reward dimension mechanism, calculate the reward signals of each reward dimension according to the current system state and the next system state to obtain a reward signal matrix. Perform a non-linear reward combination on the reward signal matrix to obtain a preliminary reward function, and calculate the dynamic discount factor of the next system state. Then, calculate the total reward function in combination with the default penalty and the preliminary reward function, and perform dynamic iterative update calculation on the value function in combination with the total reward function and the dynamic discount factor to obtain a converged value function. Perform resource adjustment on the target system in combination with this value function and the current system state. By quantifying the calculation of the default penalty based on the degree of constraint violation, the present application makes the constraint handling more flexible. It not only considers the violation of constraints but also accurately quantifies the degree of violation, improving the exploration ability of the algorithm and the quality of the solution. At the same time, using non-linear reward combination instead of fixed simple linear combination can better reflect the complex relationships and performance trade-offs between different reward dimensions, enhancing the flexibility and adaptability of the reward function. In addition, introducing a dynamic discount factor and performing iterative update can effectively balance short-term benefits and long-term decisions, ensuring that the algorithm can be optimized from a global perspective, thereby achieving continuous performance guarantee. In addition, performing resource adjustment on the target system in combination with the value function and the current system state can more effectively perform resource scheduling when the value function converges, improving the optimization effect and performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the technical solutions in the embodiments or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0046] Figure 1 It is a flowchart of a method for resource adjustment of a system provided in an embodiment of the present application;

[0047] Figure 2 It is a schematic structural diagram of a device for resource adjustment of a system provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] To facilitate understanding of the technical solutions provided in the embodiments of the present application, the following will first explain the background technology related to the embodiments of the present application.

[0049] In the field of computing resource scheduling, reinforcement learning methods have gradually become a research hotspot due to their adaptability and powerful optimization capabilities. However, current technologies mostly use heuristic rules or relatively simple reinforcement learning strategies for resource allocation, and these methods have obvious deficiencies when dealing with complex multi-dimensional optimization goals. Especially in the design of the reward function, there are three major problems in the reward function design:

[0050] First of all, multi-dimensional reward signals are usually processed in a simple linear combination manner, with fixed weights for each dimension and a lack of a dynamic adjustment mechanism, making it difficult to effectively capture the mutual relationships and cross-influences between different indicators, and thus unable to comprehensively reflect the performance trade-offs in complex optimization goals.

[0051] Secondly, in the handling of hard constraints, current methods mostly use the constraint conditions as hard boundaries of threshold limits, resulting in a serious compression of the feasible solution space and no quantitative evaluation of the degree of constraint violation. This not only limits the exploration potential of the algorithm but also reduces the quality and robustness of the final solution.

[0052] Finally, many methods overly focus on short-term immediate rewards, use a fixed discount factor to ignore the long-term impact of decisions and future state prediction, and lack the ability to perceive the dynamic changes of system states and loads. Therefore, it is difficult to achieve global optimization of the overall system and ensure long-term performance.

[0053] Therefore, resource scheduling combined with the current reward function has significant defects such as fixed weights, rigid constraint handling, and lack of a long-term perspective, and it is difficult to meet the optimization requirements of complex systems.

[0054] To solve this problem, an embodiment of the present application provides a method, device, and equipment for resource adjustment of a system. First, obtain the current state of the target system and the next system action. The current state covers key indicators such as resource utilization rate, quality of service, system energy consumption, and system cost. After executing the next system action, the system enters a new state, and combines a preset set of constraint conditions to calculate the default penalty during the state change process. This default penalty is obtained by quantifying the degree of constraint violation. Based on the reward dimension mechanism, calculate the reward signals for each dimension according to the current and next system states to form a reward signal matrix. Subsequently, process the reward signal matrix using a non-linear combination method to obtain a preliminary reward function, and at the same time calculate the dynamic discount factor for the next state. Integrate the default penalty with the preliminary reward function to calculate the total reward function, and use this to iteratively update the value function in combination with the dynamic discount factor until convergence. Adjust the resources of the target system based on the converged function and the current system state to achieve a balance between system performance optimization and constraint satisfaction. The present application adopts a non-linear reward combination method, which can better reflect the complex relationships and performance trade-offs between indicators compared to a simple fixed-weight linear combination. At the same time, by quantifying the degree of constraint violation to calculate the default penalty, it can more effectively handle hard constraints, expand the feasible solution space, and enhance the algorithm's exploration ability and the quality of the solution. In addition, introducing a dynamic discount factor and value function for iterative update, paying attention to long-term decision-making and system dynamic changes, helps to achieve global optimization and continuous performance guarantee, and avoid the problem of only focusing on short-term immediate benefits.

[0055] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0056] See Figure 1 , which is a flowchart of a method for resource adjustment of a system provided by an embodiment of the present application. As Figure 1 shown, the method for resource adjustment of the system may include steps S101-S106:

[0057] S101: Obtain the current system state and the next system action of the target system.

[0058] The key to step S101 is to comprehensively and accurately obtain the current state information of the target system and the next system action to be executed, providing a solid data basis for subsequent resource adjustment and optimization decisions.

[0059] Specifically, the current system state includes four core dimensions: resource utilization, quality of service, system energy consumption, and system cost. These metrics comprehensively reflect the operating conditions and performance of the system from different aspects.

[0060] First, resource utilization refers to the degree of use of hardware resources such as computing resources (e.g., Central Processing Unit (CPU) utilization, memory occupancy), storage resources, or network bandwidth in the system. High resource utilization means that the system can fully utilize the performance potential of existing resources and avoid resource waste. For example, in a cloud computing platform, a CPU utilization of 70% is generally considered ideal, ensuring sufficient processing power while avoiding the risk of overload.

[0061] Second, Quality of Service (QoS) reflects the system's ability to respond to user needs and stability, including metrics such as response time, task completion rate, reliability, and availability. For example, an online e-commerce platform that can ensure a 99.9% successful request response rate and a page load time of less than 200 milliseconds indicates a high quality of service.

[0062] Third, system energy consumption refers to the total amount of electrical energy consumed by the system during operation, which is crucial for energy conservation and emission reduction and reducing operating costs. In data center management, energy consumption can be measured by real-time monitoring of the power consumption metrics of servers. For example, if the overall daily energy consumption of a computer room is 1000 kilowatt-hours (kWh), reducing the energy consumption by 10% through intelligent scheduling can significantly save operating expenses.

[0063] Finally, system cost covers various expenses such as hardware procurement, maintenance, energy consumption, human resource management, and software licensing, and is an important indicator for measuring the economic benefits of the system. For example, for an enterprise operating a private cloud platform, its monthly total cost may include server depreciation costs of $5000, electricity costs of $2000, and salaries of operation and maintenance personnel of $3000, totaling about $10,000.

[0064] Meanwhile, the obtained next system action indicates the scheduling strategy or resource adjustment plan that the system plans to adopt, such as increasing the number of virtual machines, adjusting the load balancing strategy, or shutting down idle devices. These actions will directly affect the evolution of the system state. By combining the information of the current state and the next action, the system can achieve dynamic and adaptive resource management, meeting business requirements while taking into account energy conservation and cost control, and ultimately achieving the optimal balance between overall performance and economy.

[0065] S102: Execute the next system action to obtain the next system state, and calculate the default penalty during the change process from the current system state to the next system state calculation state in combination with the constraint condition set.

[0066] After performing the next system action, the system transitions from the current state to a new next system state, which reflects changes in key metrics such as resource allocation, quality of service, energy consumption, and cost. Specifically, the system first adjusts or schedules resources according to predefined action instructions, which may include increasing or decreasing the allocation of computing resources, adjusting the load balancing strategy, turning on or off some devices, etc., thereby causing corresponding changes in the system operating environment and performance metrics. After completing the action execution, the specific values and characteristics of the next system state are determined by monitoring and collecting relevant data.

[0067] At the same time, to ensure that the system continuously meets the designed constraint conditions during the adjustment process, it is necessary to evaluate the compliance during the state change process in combination with the constraint set. The constraint set usually includes multiple aspects such as resource capacity limits, latency requirements, energy consumption thresholds, budget limits, etc. These constraints are designed to ensure the security, stability, and economy of the system. During the transition from the current system state to the next system state, there may be situations where some constraints are violated, such as resource usage exceeding the maximum capacity, response time exceeding the defined range, or energy consumption exceeding the standard.

[0068] To quantify the impact of these default behaviors on the overall performance of the system, this application calculates the default penalty based on the degree of constraint violation. Specifically, instead of simply judging whether a constraint is violated, it measures the severity of the violation through a refined quantification method, such as the magnitude and frequency of exceeding the threshold, so as to more accurately reflect the negative impact of constraint violations on the system. This default penalty, as a numerical indicator, can be integrated into the subsequent reward function to prompt the algorithm to avoid or reduce constraint violations during optimization and improve the exploration ability and solution quality of the system.

[0069] In summary, by performing the next system action to obtain an updated system state and combining a detailed constraint set to calculate the default penalty in a quantitative manner during the state change process, it helps to achieve strict constraint management and scientific optimization of the system dynamic adjustment, making the resource scheduling both efficient and compliant with multiple complex limitations, and ensuring the stable and reliable operation of the system.

[0070] In one possible implementation, the calculation formula for the degree of constraint violation is: v j =max(0, g j (s t , a t ))), where j = 1, 2,..., m, and m represents the total number of constraints in the constraint set; v j represents the degree of constraint violation of the j-th constraint condition in the constraint set; s t represents the current system state; a tIndicates the next system action; g j (s t , a t ) represents the constraint function value corresponding to the constraint condition, g j (s t , a t ) is used to measure whether the constraint is satisfied for the j-th constraint condition in the set of constraint conditions after performing action a t in state s t ; g j (s t , a t ) is less than or equal to 0;

[0071] The calculation formula for the default penalty is: , where p j (v j ) = -β j · h(v j ); p j (v j ) represents the default penalty of v j ; h(v j ) represents the penalty shape function, h(v j ) includes h(v j ) = v j , h(v j ) = v j ², h(v j ) = -1, h(v j ) = log(1 + λv j ) for any one of them, λ represents the shape hyperparameter; β j is a dynamic parameter representing the adaptive penalty coefficient, β j = β j t-1 · (1 + δ · sign( - ε j )), δ represents the adjustment step size, β j t-1 represents the adaptive penalty coefficient during the previous iteration update, represents the average value of the constraint violation degree during the previous iteration update process; ε j represents the violation threshold.

[0072] In a possible implementation, the penalty function is selected as:

[0073] The SLA constraint adopts exponential penalty: ; The initial penalty coefficient β1 = 5.0, the shape parameter λ1 = 2.0; the violation threshold ε1 = 0.05.

[0074] The resource capacity constraint adopts quadratic penalty: p2(v2) = -β 2· v2 2 ; The initial penalty coefficient β2 = 10.0; The violation threshold ε2 = 0.02.

[0075] The energy consumption constraint adopts linear penalty: p3(v3) = -β 3· v3; The initial penalty coefficient β3 = 3.0; The violation threshold ε3 = 0.1.

[0076] In a possible implementation, the set of constraint conditions includes but is not limited to:

[0077] (1) Service Level Agreement (SLA) constraint:

[0078] Ordinary tasks: Response time ≤ 200ms;

[0079] High-priority tasks: Response time ≤ 100ms;

[0080] Critical tasks: Response time ≤ 50ms.

[0081] (2) Resource capacity constraint:

[0082] The number of CPU cores ≤ the total number of available cores;

[0083] Memory allocation ≤ the total amount of available memory;

[0084] Storage allocation ≤ the total amount of available storage;

[0085] (3) Energy consumption upper limit constraint:

[0086] Power consumption per unit time ≤ the upper limit of the computer room power;

[0087] Carbon emissions ≤ carbon emission quota.

[0088] Generally speaking, this step ensures that before any system operation is executed, its impact on the system state can be accurately evaluated, and potential risks can be controlled and managed by calculating default penalties, so as to achieve the efficient and stable operation of the system.

[0089] S103: Combining the reward dimension mechanism, calculate the reward signals of each reward dimension according to the current system state and the next system state to obtain a reward signal matrix.

[0090] During the system resource adjustment process, integrating the reward dimension mechanism is a key means to achieve multi-objective optimization. This mechanism deeply compares and analyzes the current system state with the next system state obtained after executing the next system action, calculates reward signals in multiple different dimensions, and then forms a structured reward signal matrix, thereby comprehensively reflecting the changes in various aspects of the system performance.

[0091] Specifically, the system operation involves multiple important indicators, such as resource utilization rate, service quality, system energy consumption, and system cost, etc. These indicators each represent different aspects of the system performance. Therefore, optimizing a single indicator often fails to meet complex practical requirements. Introducing the reward dimension mechanism, the above-mentioned multiple indicators can be used as different reward dimensions respectively. By calculating the corresponding reward signals according to the performance of each dimension, it helps the system to weigh various performance goals during resource adjustment and achieve comprehensive optimization.

[0092] The calculation process usually depends on the difference between the current system state and the next system state. For example, in the dimension of resource utilization rate, if the next system state shows that the resource usage is more reasonable and the overall utilization efficiency is improved, the corresponding reward signal is positive; on the contrary, if there is resource waste or an overload risk, the reward signal may be negative. In the dimension of service quality, if the response time is shortened, the task success rate is increased, or the system stability is enhanced, positive reward signals will also be generated. The system energy consumption dimension focuses on whether the energy consumption is reduced: if the next system state shows that the energy consumption is decreased, the reward signal is positive; if the energy consumption is increased or exceeds the preset threshold, the reward signal is negative.

[0093] The system cost dimension is also very important, which reflects the impact of resource adjustment on the economic cost. If the resource adjustment reduces the operating cost of the system, such as reducing the use of hardware devices or optimizing the software configuration, the corresponding reward signal should be positive; on the contrary, if the resource adjustment leads to an increase in cost, the reward signal may be negative. In this way, the system can comprehensively evaluate the effect of each resource adjustment and make more reasonable and effective decisions based on this, achieving multi-dimensional optimization and balance.

[0094] To sum up, the reward dimension mechanism not only enables the system to carefully evaluate the impact of various operations, but also can dynamically adjust and optimize the system performance during the resource adjustment process, thus ensuring that the system maintains efficient and stable operation while meeting multiple optimization goals.

[0095] Exemplarily, assume there is a cloud computing platform that needs to dynamically allocate computing resources to meet the needs of different services. The system states and actions are as follows:

[0096] System Status (Current Status): Resource Utilization: 80%, Quality of Service Metrics (such as latency): 50ms, Energy Consumption: 1000W, Cost: $20 / hour. Action (Decision Made): The CPU resources allocated to a certain computing service are increased by 20%.

[0097] After executing this action, the system observes the following new status information: The new resource utilization becomes 85%, the new quality of service metrics (such as latency) becomes 45ms, the new energy consumption becomes 950W, and the new cost becomes $19 / hour.

[0098] To convert these state changes into a reward signal, a reward function for each dimension needs to be defined. Suppose there is the following simple reward dimension mechanism:

[0099] Resource Utilization r1: If the resource utilization increases, the reward increases; otherwise, the reward decreases.

[0100] Quality of Service r2: If the quality of service improves (e.g., latency decreases), the reward increases; otherwise, the reward decreases.

[0101] Energy Consumption r3: If the energy consumption decreases, the reward increases; otherwise, the reward decreases.

[0102] Cost r4: If the cost decreases, the reward increases; otherwise, the reward decreases.

[0103] Then the specific calculation process of the reward signal matrix is as follows:

[0104] Resource Utilization Reward r1: The current utilization increases from 80% to 85%, an increase of 5%. Suppose the reward function is to reward 1 point for every 1% increase in resource utilization: r1 = 5×1 = 5;

[0105] Quality of Service Reward r2: The latency decreases from 50ms to 45ms, a decrease of 5ms. Suppose a reward of 2 points is given for every 1ms decrease in latency: r2 = 5×2 = 10;

[0106] Energy Consumption Reward r3: The energy consumption decreases from 1000W to 950W, a decrease of 50W. Suppose a reward of 0.1 point is given for every 1W decrease in energy consumption: r3 = 50×0.1 = 5;

[0107] Cost Reward r4: The cost decreases from $20 / hour to $19 / hour, a decrease of $1 / hour. Suppose a reward of 3 points is given for every $1 / hour decrease in cost: r4 = 1×3 = 3.

[0108] Combine these reward signals into a multi-dimensional raw reward signal (r1, r2, r3, r4):

[0109] (r1, r2, r3, r4) = (5, 10, 5, 3).

[0110] Through the above steps, the state changes and actions of the system can be converted into quantifiable multi-dimensional reward signals. These reward signals are used to evaluate the effects of each action and guide the subsequent decision-making process. This mechanism can help the system continuously optimize resource allocation and achieve the balance and optimization of multi-dimensional goals.

[0111] In a possible implementation, the reward dimension mechanism includes but is not limited to:

[0112] : Resource utilization reward, measuring the utilization efficiency of resources such as CPU and memory:

[0113] : Reward when the CPU utilization does not exceed the target range [60%, 85%];

[0114] : Reward when the memory utilization does not exceed the target range [55%, 80%];

[0115] : Reward when the storage utilization does not exceed the target range [50%, 75%].

[0116] : Service quality reward, measuring service metrics such as response time and throughput:

[0117] : Reward when the response time < 100 ms;

[0118] : Reward when the throughput > 1000 requests / second;

[0119] : Reward when the error rate < 0.1%.

[0120] : Energy consumption reward, measuring energy consumption and carbon emissions:

[0121] : Reward when the power efficiency > 80%;

[0122] : Reward when the carbon emission reward < the set threshold.

[0123] : Economic cost reward, measuring operating costs and resource waste:

[0124] : Reward when the resource cost < the budget limit;

[0125] : When the resource waste award < 10%, give a reward.

[0126] S104: Perform a non - linear reward combination on the said reward signal matrix to obtain a preliminary reward function, and calculate the dynamic discount factor of the said next system state.

[0127] In the actual system optimization process, simply relying on the independent evaluation of each reward dimension often cannot comprehensively reflect the complex correlation of the overall system performance. Therefore, it is necessary to perform a non - linear combination on the reward signal matrix to obtain a more representative comprehensive reward function. Specifically, by introducing a non - linear combination method, the interaction effects and weight differences between different reward dimensions can be captured, thus avoiding problems such as information loss or index conflict that may be brought about by simple weighted summation. For example, when there is a positive or negative correlation between different dimensions, the non - linear function can appropriately amplify or suppress the influence of some reward signals, making the finally obtained preliminary reward function more in line with the true needs of the system goal.

[0128] At the same time, the dynamic discount factor γ(s) corresponding to the next system state can be calculated, which is used to measure the value weight of future rewards in the current state. The calculation formula of this dynamic discount factor is.

[0129] γ(s)=γbase + △γ(s);

[0130] Among them, γbase represents the base discount factor, which is a dynamic parameter used to ensure that the discount factor fluctuates within a reasonable range; while △γ(s) is the discount term dynamically adjusted according to the system state characteristics, and it is calculated in the following way:

[0131] △γ(s)=ω1·φ1(load(s)) + ω2·φ2(scarcity(s)) + ω3·φ3(urgency(s)). Here, the system load (load(s)), resource scarcity (scarcity), and task urgency (urgency) are important factors affecting the discount adjustment. They are respectively converted into values available for discount adjustment through the mapping function φi(·). Specifically, the mapping function maps these original indicators to a standardized discount adjustment range, ensuring that their contributions to the discount factor are both scientifically based and easy to adjust. The preset weight coefficient ωi is used to control the relative importance of each factor in the discount adjustment. ωi is a dynamic parameter that can achieve flexible adaptation to different operating environments and business scenarios.

[0132] In a possible implementation, the dynamic discount setting:

[0133] The base discount factor γbase = 0.9;

[0134] System load mapping function: φ1(load) = 0.05·tanh(2·load - 1); when the load is below 30%, the discount decreases, and more attention is paid to short-term benefits. Resources can be allocated more liberally to improve the user experience; when the load is above 70%, the discount increases, and more attention is paid to long-term planning, reserving some capacity to handle possible critical tasks in the future.

[0135] Resource scarcity mapping function: φ2(scarcity) = 0.03·scarcity; when resources are sufficient (scarcity < 0.3), the discount decreases. When there are a large number of idle GPUs in the data center, more resources can be allocated to AI training tasks to improve the current training speed; when resources are tense (scarcity > 0.7), the discount increases. When resources are close to exhaustion, the system needs to use resources more sparingly and give priority to ensuring critical services.

[0136] Task urgency mapping function: φ3(urgency) = -0.04·urgency; when there are many urgent tasks (urgency > 0.7), the discount decreases. When the system faces a large number of urgent tasks, the current needs should be met first, and the future should not be over-considered; when there are few urgent tasks (urgency < 0.3), the discount increases. For example, during the night batch processing period, the system can reserve sufficient resources for long-running training tasks to improve the overall throughput.

[0137] In summary, by combining the non-linear reward combination with the calculation of the dynamic discount factor, not only can the initial reward function more accurately reflect the multi-dimensional system performance, but also the weight of future rewards can be adjusted in real time through the dynamic discount mechanism, taking into account the balance between the current state and the long-term goal, thereby significantly improving the intelligence and robustness of the system resource adjustment strategy.

[0138] In a possible implementation, step S104 performs non-linear reward combination on the reward signal matrix to obtain the initial reward function, including A1 - A2:

[0139] A1: Construct the correlation matrix of the reward signal matrix, and normalize each dimension of the reward in the reward signal matrix to obtain the normalized reward matrix.

[0140] First, according to the correlation relationship between each reward dimension in the reward signal matrix, construct the correlation matrix. The diagonal elements of this correlation matrix are all 1, indicating that each reward dimension is completely positively correlated with itself; the non-diagonal elements reflect the correlation degree between the i-th index and the j-th index, and the value range is [-1, 1], which is used to describe the positive and negative correlation strengths and directions between different indicators. At the same time, perform normalization processing on each dimension of the reward in the reward signal matrix to eliminate the differences in the dimensions and value ranges of different indicators, so as to obtain the normalized reward matrix, laying a foundation for subsequent combination calculations.

[0141] Wherein, the correlation matrix is: .

[0142] =1 specifies that all elements on the matrix diagonal are equal to 1, indicating that each reward dimension is completely positively correlated with itself. For example, when CPU utilization increases, the CPU utilization reward will inevitably change accordingly. This is a deterministic relationship that ensures that each dimension's contribution is fully considered when calculating the total reward.

[0143] Indicates the degree of correlation between the i-th indicator and the j-th indicator, ∈[-1,1] limits the range of values for off-diagonal elements and characterizes the strength and direction of the correlation between different dimensions. A positive value (e.g., 0.5) indicates that the two dimensions are moving in the same direction. For example, when energy consumption increases, economic costs tend to increase as well. A negative value (e.g., -0.7) indicates that the two dimensions are moving in opposite directions. For example, increasing resource utilization tends to reduce service quality. Values close to zero indicate that the two dimensions are nearly independent and have no influence on each other. The magnitude of the value indicates the strength of the correlation: |0.9| indicates a stronger correlation than |0.3|.

[0144] = , ensuring matrix symmetry, dimension i's impact on dimension j is the same as dimension j's impact on dimension i. For example, the impact of CPU usage on memory usage and the impact of memory usage on CPU usage are treated as equal when calculating rewards. This symmetry simplifies the mathematical model, ensures computational stability, and avoids decision bias.

[0145] In a possible implementation, the correlation matrix can be specifically set as:

[0146]

[0147] This matrix indicates that resource utilization is negatively correlated with service quality (-0.7) and positively correlated with energy consumption (0.5); energy consumption is positively correlated with economic costs (0.6), etc. The construction of the correlation matrix is based on the following data sources: historical operational data analysis: correlation analysis of system operation data from the past six months; expert experience input: evaluation results from five domain experts; and online adaptive adjustment: fine-tuning the system during operation based on real-time feedback.

[0148] In one possible implementation, the present application uses an adaptive normalization method to normalize the rewards of each dimension in the reward signal matrix. The adaptive normalization formula is: .

[0149] Among them, μ iand σ i are the mean and standard deviation of dimension i, respectively, which are dynamically updated by the sliding window method.

[0150] A2: Using a non - linear combination function to combine the correlation matrix, combine the rewards of each dimension in the normalized reward matrix to obtain a preliminary reward function.

[0151] Based on the normalized reward matrix and the correlation matrix, a non - linear combination function is used to organically integrate the rewards of each dimension to form a preliminary reward function. The preliminary reward function not only includes the linear weighted terms of each reward dimension, but also introduces a cross - term function to capture the mutual influence and synergy between different dimensions.

[0152] In a possible implementation, the non - linear combination function is: , where R is the preliminary reward function; and represent the rewards in the reward signal matrix; represents the cross - term function; w i is a dynamic parameter, representing the adaptive weight of r i .

[0153] It should be noted that the weight w of each reward dimension i is a dynamic adaptive parameter, which is continuously adjusted according to the system performance feedback. The weight update formula is: w i = w i （t-1） ·(1 + η·sign(△Pi)·|△Pi| k ).

[0154] Among them, w i （t-1） represents the weight value in the previous iteration; η is a dynamic parameter, which is a learning rate control parameter and determines the weight adjustment amplitude; sign(△Pi) is a sign function, which is used to indicate the adjustment direction (increase or decrease); k is a dynamic sensitivity parameter, which adjusts the non - linear degree of the adjustment; the performance gap ΔPi is: △Pi=(Pi - Ti) / Ti, where Pi is the actual performance value of the corresponding reward dimension in the current system state, and Ti is the target performance value of this index. Through this mechanism, the weight can be intelligently corrected according to the deviation between the actual and the target, so that the combined reward function can more accurately reflect the current performance of the system and the optimization requirements.

[0155] In a possible implementation, the initial weights are set as: w1 = 0.3, w2 = 0.3, w3 = 0.2, w4 = 0.2; the learning rate: η = 0.05; the sensitivity parameter: k = 0.5.

[0156] It should also be noted that is a cross - term function used to model dimensions and dimension 's interaction effect. The cross - term function is designed considering the following forms:

[0157] Product form: ;

[0158] Exponential form: ;

[0159] Threshold form: ,

[0160] where α is the cross - intensity parameter, θ i and θ j are threshold parameters.

[0161] In a possible implementation, to ensure that the sum of weights is 1, it is necessary to normalize the adaptive weight w i for any r i : .

[0162] In a possible implementation, to avoid drastic fluctuations in weights, the adaptive weight w i can be smoothed to obtain a smoothed adaptive weight w i ´, and the smoothed adaptive weight w i ´ is used to replace the adaptive weight w i to calculate the preliminary reward function R. w i ´=(1 - ρ)·w i +ρ·w i （t-1） , where ρ represents the smoothing coefficient for controlling the weight smoothness, and the smoothing coefficient ρ = 0.8.

[0163] S105: Combine the default penalty and the preliminary reward function to calculate the total reward function to obtain the total reward function, and combine the default penalty and the preliminary reward function to calculate the total reward function, and combine the total reward function and the dynamic discount factor of the next system state to perform dynamic iterative update calculation on the value function until a convergent value function is obtained.

[0164] Combining the default penalty and the preliminary reward function, the total reward function of the system can be calculated. Specifically, the total reward function R total is composed of the preliminary reward function R and the sum of default penalty terms under all constraint conditions, and its calculation formula is:

[0165] R total =R + ;

[0166] where R is the preliminary reward function; For the said default penalty, m represents the total number of constraints in the set of constraints. Such a design ensures the performance of the reward system while effectively considering and punishing the system's behavior of violating constraints, and promotes the resource adjustment process to meet relevant restrictions.

[0167] Based on this total reward function and the dynamic discount factor γ(s t+1 ) corresponding to the next system state, the value function Q can be updated iteratively to continuously optimize the strategy. Its update formula is:

[0168] Q = Q(s t , a t ) + α[R total + γ(s t+1 ) · max α Q(s t+1 , α) - Q(s t , a t )]

[0169] Where s t represents the current system state; s t+1 represents the next system state obtained after executing the next system action a t [[ID=3,4]]; Q(s t , a t ) represents the expected cumulative reward value of executing the next system action at in the current system state s t ; α is a dynamic parameter representing the learning rate; R total represents the said total reward function; γ(s t+1 ) represents the dynamic discount factor of the next system state; max α Q(s t+1 , α) represents the maximum value of all possible actions in the next system state s t+1 .

[0170] Through the above iterative update process, the value function is gradually adjusted, enabling the system to more accurately evaluate the long-term benefits and risk trade-offs brought by different actions. When the value function converges, it indicates that the strategy tends to be stable. At this time, the target system can be effectively resource-adjusted based on this value function to achieve comprehensive optimization of system performance and constraints.

[0171] In a possible implementation, in step S105, the total reward function is calculated by combining the default penalty and the preliminary reward function to obtain the total reward function, and the value function is dynamically iteratively updated and calculated by combining the total reward function and the dynamic discount factor of the next system state until a convergent value function is obtained, including:

[0172] The value function is calculated by using the dynamic iterative update calculation formula, in combination with the total reward function and the dynamic discount factor of the next system state. If the value function does not converge, the dynamic parameters and hyperparameters in the formulas used to calculate the preliminary reward function, the dynamic discount factor, and the value function are iteratively adjusted until the converged value function is obtained and then the iteration stops.

[0173] Specifically, first, according to the dynamic iterative update calculation formula, the value function is calculated by using the current total reward function and the dynamic discount factor of the next system state to obtain the latest value function; then it is judged whether this value function has converged. If the convergence condition is not met, the system will adjust and optimize various dynamic parameters and hyperparameters used in the calculation of the preliminary reward function, the dynamic discount factor, and the value function, so as to obtain a converged value function by using the dynamic iterative update calculation formula in combination with the new total reward function and the dynamic discount factor of the new next system state.

[0174] This process is repeatedly executed until the latest value function meets the convergence requirements and the iteration stops. Through this closed-loop adaptive iteration mechanism, the stability and accuracy of the value function can be effectively improved, thereby providing a reliable basis for subsequent resource adjustment strategies and improving the overall system performance and constraint satisfaction.

[0175] S106: Adjust the resources of the target system in combination with the converged value function and the current system state.

[0176] After obtaining the value function, the resources of the target system can be adjusted in combination with the value function to utilize the state-action value evaluation reflected by this function and select the optimal resource allocation strategy in a targeted manner. Through the weighing and prediction of future rewards by the value function, the key elements such as the computing resources, storage resources, and network bandwidth of the target system can be reasonably adjusted to improve overall performance indicators such as response speed, energy consumption efficiency, and service quality. At the same time, this adjustment process comprehensively considers the current system state and constraint conditions to ensure the effectiveness and stability of resource use, thereby realizing the dynamic optimization and adaptive management of resource allocation and promoting the efficient operation of the target system in a complex environment.

[0177] In a possible implementation manner, the adjusting the resources of the target system in combination with the converged value function and the current system state includes:

[0178] Outputting the state-action value according to the converged value function and the current system state, and adjusting the resources of the target system in combination with the state-action value.

[0179] The specific process includes: First, input the current system state into the already converged value function. Based on this, the value function calculates and outputs the state-action value corresponding to the executable action at the next moment in this state. Subsequently, based on this state-action value, resource adjustment is performed on the target system to achieve dynamic optimization of resource allocation. In this way, the resource adjustment strategy not only makes full use of the accurate estimation of the value function for future benefits but also ensures that each adjustment tends to improve the overall system performance and efficiency, with strong real-time adaptability and optimization effects.

[0180] Based on the content of S101 - S106, it starts by collecting the current system state of the target system and the predicted next system action. The current system state covers key indicators such as resource utilization rate, service quality, system energy consumption, and system cost. After executing the predicted next system action, a new system state is obtained, and at the same time, any constraint violations occurring during the transition from the current state to the next state are quantified based on a set of constraint condition sets to calculate the default penalty. This default penalty is quantitatively analyzed based on the degree of violation. Next, through the reward dimension mechanism, the reward signals for each dimension are calculated based on the current system state and the next system state, and finally, a reward signal matrix is formed. Then, a non-linear reward combination strategy is applied to this reward signal matrix to generate a preliminary reward function and calculate the dynamic discount factor for the next system state. On this basis, the total reward function is calculated by combining the default penalty and the preliminary reward function, and then the value function is dynamically iteratively updated by combining the total reward function and the dynamic discount factor until the value function converges. When this convergence state is reached, the target system is adjusted using the finally determined value function to achieve optimal resource allocation and efficiency improvement. In this application, by quantitatively calculating the default penalty based on the degree of constraint violation, the constraint handling is made more flexible. It not only considers the constraint violations but also accurately quantifies the degree of violation, improving the exploration ability of the algorithm and the quality of the solution. At the same time, using a non-linear reward combination to replace the fixed simple linear combination better reflects the complex relationships and performance trade-offs between different reward dimensions, enhancing the flexibility and adaptability of the reward function. In addition, introducing a dynamic discount factor and performing iterative updates effectively balances short-term benefits and long-term decisions, ensuring that the algorithm can be optimized from a global perspective and achieve continuous performance guarantee.

[0181] See Figure 2 , Figure 2 is a schematic structural diagram of a resource adjustment device for a system provided by an embodiment of this application. As Figure 2 shown, the resource adjustment device of this system includes:

[0182] An acquisition unit 201, configured to acquire the current system state and the next system action of the target system; the current system state includes the resource utilization rate, service quality, system energy consumption, and system cost of the target system;

[0183] An execution unit 202, configured to execute the next system action to obtain the next system state;

[0184] A default penalty calculation unit 203, configured to calculate the default penalty during the change process from the current system state to the next system state calculation state in combination with a set of constraint conditions; the default penalty is quantitatively calculated based on the degree of constraint violation during the change process from the current system state to the next system state calculation state;

[0185] A reward signal matrix calculation unit 204, configured to calculate the reward signals of each reward dimension according to the current system state and the next system state in combination with a reward dimension mechanism, and obtain a reward signal matrix;

[0186] A preliminary reward function calculation unit 205, configured to perform a non-linear reward combination on the reward signal matrix to obtain a preliminary reward function;

[0187] A dynamic discount factor calculation unit 206, configured to calculate the dynamic discount factor of the next system state;

[0188] A total reward function calculation unit 207, configured to calculate a total reward function by combining the default penalty and the preliminary reward function;

[0189] A value function calculation unit 208, configured to calculate a total reward function by combining the default penalty and the preliminary reward function, and perform dynamic iterative update calculation on the value function in combination with the total reward function and the dynamic discount factor of the next system state until a convergent value function is obtained;

[0190] A resource adjustment unit 209, configured to perform resource adjustment on the target system in combination with the convergent value function and the current system state.

[0191] In a possible implementation manner, the value function calculation unit 208 is specifically configured to:

[0192] Calculate a value function by using the dynamic iterative update calculation formula of the value function and combining the total reward function and the dynamic discount factor of the next system state;

[0193] If the value function does not converge, iteratively adjust and calculate the dynamic parameters and hyperparameters in the formulas used for calculating the preliminary reward function, the dynamic discount factor, and the value function until the convergent value function is obtained and then stop.

[0194] In a possible implementation, the calculation formula for the degree of constraint violation is: v j = max(0, g j (s t , a t )), where j = 1, 2,..., m, and m represents the total number of constraints in the constraint set; v j represents the degree of constraint violation of the j-th constraint condition in the constraint set; s t represents the current system state; a t represents the next system action; g j (s t , a t ) represents the constraint function value of the corresponding constraint condition, and g j (s t , a t ) is used to measure whether the constraint is satisfied for the j-th constraint condition in the constraint set after performing action a t in state s t , and g j (s t , a t ) is less than or equal to 0;

[0195] The calculation formula for the default penalty is: , where p j (v j ) = -β j · h(v j ); p j (v j ) represents the default penalty of v j ; h(v j ) represents the penalty shape function, and h(v j ) includes h(v j ) = v j , h(v j ) = v<� j ², h(v j ) = -1, h(v j ) = log(1 + λv j ) of any one, λ represents the shape hyperparameter; β j is a dynamic parameter representing the adaptive penalty coefficient, β j = β j t-1 · (1 + δ · sign( - ε j )), δ represents the adjustment step size, and β j t-1 represents the adaptive penalty coefficient at the previous iteration update, represents the average degree of constraint violation in the previous iteration update process; ε j represents the violation threshold.

[0196] In a possible implementation, obtaining the preliminary reward function by performing non - linear reward combination on the reward signal matrix includes:

[0197] Construct the correlation matrix of the reward signal matrix, and normalize each dimension reward in the reward signal matrix to obtain the normalized reward matrix;

[0198] Use the non - linear combination function to combine with the correlation matrix, and combine each dimension reward in the normalized reward matrix to obtain the preliminary reward function;

[0199] wherein, the correlation matrix is: , = 1, represents that each reward dimension is completely positively correlated with itself; represents the correlation degree between the i - th index and the j - th index, ∈[-1,1]; = ;

[0200] The non - linear combination function is: , R is the preliminary reward function; and represents the reward in the reward signal matrix; represents the cross - term function; w i is a dynamic parameter, representing r i 's adaptive weight; w i = w i （t-1） ·(1 + η·sign(△Pi)·|△Pi| k ); wi （t -1） represents the weight at the previous iteration update ; η is a dynamic parameter, representing the learning rate; sign(△Pi) represents the sign function that determines whether to adjust and the adjustment direction according to △Pi; k is a dynamic parameter, representing the adjustment sensitivity parameter; △Pi represents the performance gap, △Pi=(Pi - Ti) / Ti, Pi represents the current system state used when calculating the reward in the reward signal matrix ; Ti represents the target state of Pi.

[0201] In a possible implementation, the calculation formula of the dynamic discount factor is: γ(s)=γ base +△γ(s);

[0202] Among them, γ(s) represents the dynamic discount factor of the system state; γ base is a dynamic parameter representing the basic discount factor; △γ(s) represents the discount adjustment term of the system state, and △γ(s)=ω1·φ1(load(s))+ω2·φ2(scarcity(s))+ω3·φ3(urgency(s)); load(s) represents the system load influencing factor, scarcity(s) represents the resource scarcity influencing factor; urgency(s) represents the task urgency influencing factor; φi(·) is a mapping function that maps each factor to the discount adjustment value; ωi is a dynamic parameter representing the preset weight coefficient; the value of i is 1, 2, 3.

[0203] In a possible implementation manner, the calculation formula of the total reward function is: R total =R + ;

[0204] Among them, R is the preliminary reward function; is the default penalty, and m represents the total number of constraints in the constraint set.

[0205] In a possible implementation manner, the dynamic iterative update calculation formula of the value function is: Q = Q(s t , a t ) + α[R total +γ(s t+1 )·max α Q(s t+1 , α) - Q(s t , a t )];

[0206] Among them, s t represents the current system state; s t+1 represents the next system state obtained after executing the next system action a t ; Q(s t , a t ) represents the expected cumulative reward value of executing the next system action at in the current system state s t ; α is a dynamic parameter representing the learning rate; R total represents the total reward function; γ(s t+1 ) represents the dynamic discount factor of the next system state; max α Q(s t+1 , α) represents the maximum value of all possible actions in the next system state s t+1 .

[0207] In a possible implementation manner, the device further includes:

[0208] A smoothing unit for smoothing the adaptive weight w i to obtain a smoothed adaptive weight w i ´, and using the smoothed adaptive weight w i ´ to replace the adaptive weight w i to calculate the preliminary reward function R;

[0209] The smoothed adaptive weight w i ´ = (1 - ρ) · w i + ρ · w i （t-1） , where ρ represents a smoothing coefficient for controlling the weight smoothness.

[0210] In addition, an embodiment of the present application further provides a system resource adjustment device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the system resource adjustment method described above is implemented.

[0211] The present application realizes flexible adjustment of constraint handling by quantifying and calculating default penalties based on the degree of constraint violation, taking into account both whether the constraint is violated and accurately measuring the severity of the violation, thereby improving the exploration ability and solution quality of the algorithm. At the same time, using non-linear reward combination to replace fixed linear weighting can more realistically reflect the complex relationship and performance trade-off between reward dimensions, enhancing the expression ability and adaptability of the reward function. Furthermore, introducing a dynamic discount factor and updating it iteratively can effectively balance short-term benefits and long-term decisions, enabling the algorithm to optimize resource scheduling from a global perspective and ensuring the continuous and stable performance of the system.

[0212] The above has introduced in detail a system resource adjustment method, device, and equipment provided by the present application. The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0213] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item) of the following" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

Claims

1. A method for resource adjustment of a system, characterized in that The method includes: Obtaining the current system state and the next system action of the target system; the current system state includes the resource utilization rate, service quality, system energy consumption, and system cost of the target system; Performing the next system action to obtain the next system state, and calculating the default penalty during the change process from the current system state to the next system state calculation state in combination with the constraint condition set; the default penalty is quantitatively calculated based on the degree of constraint violation during the change process from the current system state to the next system state calculation state; Combining the reward dimension mechanism, and calculating the reward signals of each reward dimension according to the current system state and the next system state to obtain a reward signal matrix; Performing a non-linear reward combination on the reward signal matrix to obtain a preliminary reward function, and calculating the dynamic discount factor of the next system state; Calculating the total reward function by combining the default penalty and the preliminary reward function, and dynamically iteratively updating and calculating the value function by combining the total reward function and the dynamic discount factor of the next system state until a converged value function is obtained; Adjusting the resources of the target system by combining the converged value function and the current system state.

2. The method according to claim 1, wherein The calculating the total reward function by combining the default penalty and the preliminary reward function, and dynamically iteratively updating and calculating the value function by combining the total reward function and the dynamic discount factor of the next system state until a converged value function is obtained includes: Calculating the value function by using the dynamic iterative update calculation formula of the value function and combining the total reward function and the dynamic discount factor of the next system state; If the value function does not converge, iteratively adjust the dynamic parameters and hyperparameters in the formulas used to calculate the preliminary reward function, the dynamic discount factor, and the value function until the converged value function is obtained and then stop.

3. The method according to claim 1, characterized in that, The calculation formula for the degree of constraint violation is: v j = max(0, g j (s t , a t ))), where j = 1, 2,..., m, and m represents the total number of constraints in the set of constraint conditions; v j represents the degree of constraint violation of the j-th constraint condition in the set of constraint conditions; s t represents the current system state; a t represents the next system action; g j (s t , a t ) represents the constraint function value of the corresponding constraint condition, and g j (s t , a t ) is used to measure whether the constraint is satisfied for the j-th constraint condition in the set of constraint conditions after performing the action a t in the state s t , and g j (s t , a t ) is less than or equal to 0; The calculation formula for the default penalty is as follows: , where p j (v j ) = -β j · h(v j ); p j (v j ) represents the default penalty of v j ; h(v j ) represents the penalty shape function, and h(v j ) includes any one of h(v j ) = v j , h(v j ) = v j ², h(v j ) = -1, h(v j ) = log(1 + λv j ), where λ represents the shape hyperparameter; β j is a dynamic parameter representing the adaptive penalty coefficient, and β j = β j t-1 · (1 + δ · sign( - ε j ), where δ represents the adjustment step size, β j t-1 represents the adaptive penalty coefficient at the previous iteration update, represents the average value of the constraint violation degree during the previous iteration update; ε j represents the violation threshold.

4. The method according to claim 1, characterized in that The performing a non-linear reward combination on the reward signal matrix to obtain a preliminary reward function includes: Constructing a correlation matrix of the reward signal matrix, and normalizing the rewards of each dimension in the reward signal matrix to obtain a normalized reward matrix; Combining the rewards of each dimension in the normalized reward matrix by using a non-linear combination function in combination with the correlation matrix to obtain a preliminary reward function; Among them, the correlation matrix is as follows: , = 1, indicating that each reward dimension is completely positively correlated with itself; indicating the degree of correlation between the i-th index and the j-th index, ∈ [-1, 1]; = ; The non-linear combination function is as follows: , where R is the preliminary reward function; and represents the reward in the reward signal matrix; represents the cross-term function; w i is a dynamic parameter, representing r i 's adaptive weight; w i = w i （t-1） · (1 + η · sign(△Pi) · |△Pi| k ); w i （t-1） represents the weight at the last iteration update ; η is a dynamic parameter, representing the learning rate; sign(△Pi) represents the sign function that determines whether to adjust and the adjustment direction according to △Pi; k is a dynamic parameter, representing the adjustment sensitivity parameter; △Pi represents the performance gap, △Pi = (Pi - Ti) / Ti, where Pi represents the current system state used when calculating the reward in the reward signal matrix ; Ti represents the target state of Pi.

5. The method according to claim 1, wherein The calculation formula for the dynamic discount factor is: γ(s) = γ base + △γ(s); Among them, γ(s) represents the dynamic discount factor of the system state; γ base is a dynamic parameter representing the basic discount factor; △γ(s) represents the discount adjustment term of the system state, and △γ(s) = ω1·φ1(load(s)) + ω2·φ2(scarcity(s)) + ω3·φ3(urgency(s)); load(s) represents the system load influencing factor, scarcity(s) represents the resource scarcity influencing factor; urgency(s) represents the task urgency influencing factor; φi(·) is a mapping function that maps each factor to the discount adjustment value; ωi is a dynamic parameter representing the preset weight coefficient; the value of i is 1, 2, or 3.

6. The method according to claim 1, wherein The calculation formula of the total reward function is: R total = R + ; wherein, R is the preliminary reward function; is the default penalty, and m represents the total number of constraints in the constraint set.

7. The method according to claim 1 or 2, characterized in that, The dynamic iterative update calculation formula for the value function is: Q = Q(s t , a t ) + α[R total + γ(s t+1 )·max α Q(s t+1 , α) - Q(s t , a t )]; Among them, s t represents the current system state; s t+1 represents the next system state obtained after executing the next system action a t ; Q(s t , a t ) represents the expected cumulative reward value of executing the next system action a t in the current system state s t ; α is a dynamic parameter representing the learning rate; R total represents the total reward function; γ(s t+1 ) represents the dynamic discount factor of the next system state; max α Q(s t+1 , α) represents the maximum value of all possible actions in the next system state s t+1 .

8. The method according to claim 4, wherein The method further includes: Smoothing the adaptive weight w i to obtain a smoothed adaptive weight w i ´, and using the smoothed adaptive weight w i ´ to replace the adaptive weight w i for calculating a preliminary reward function R; The smooth adaptive weight w i ´ = (1 - ρ) · w i + ρ · w i （t-1） , where ρ represents a smoothing coefficient for controlling the smoothness of the weight.

9. A resource adjustment device for a system, characterized in that The device includes: An obtaining unit, configured to obtain the current system state and the next system action of the target system; the current system state includes the resource utilization rate, service quality, system energy consumption, and system cost of the target system; An execution unit, configured to perform the next system action to obtain the next system state; A default penalty calculation unit, configured to calculate the default penalty during the change process from the current system state to the next system state calculation state in combination with the constraint condition set; the default penalty is quantitatively calculated based on the degree of constraint violation during the change process from the current system state to the next system state calculation state; A reward signal matrix calculation unit, configured to calculate reward signals for each reward dimension according to the current system state and the next system state in combination with a reward dimension mechanism, so as to obtain a reward signal matrix; A preliminary reward function calculation unit, configured to perform a non-linear reward combination on the reward signal matrix to obtain a preliminary reward function; A dynamic discount factor calculation unit, configured to calculate a dynamic discount factor of the next system state; A total reward function calculation unit, configured to calculate a total reward function by combining the default penalty and the preliminary reward function; A value function calculation unit, configured to calculate a total reward function by combining the default penalty and the preliminary reward function, and perform dynamic iterative update calculation on the value function by combining the total reward function and the dynamic discount factor of the next system state until a converged value function is obtained; A resource adjustment unit, configured to perform resource adjustment on the target system by combining the converged value function and the current system state.

10. A resource adjustment device for a system, characterized in that Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, a resource adjustment method of the system according to any one of claims 1-8 is implemented.

Images