A resource adjustment method, device and equipment of a system

Abstract

The application discloses a resource adjustment method, device and equipment of a system. First, the current state and the next action of the target system can be acquired, the next state is obtained after the action is executed, and the default penalty and the multi-dimensional reward signal matrix in the process of changing from the current state to the next state are calculated. The reward signal matrix is combined to form a preliminary reward function, and the dynamic discount factor of the next state is calculated. The total reward function is constructed by combining the default penalty and the preliminary reward function, the value function is iteratively updated based on the reward function and the dynamic discount factor until convergence. Finally, the target system is effectively adjusted according to the converged value function. The application realizes flexible default penalty by quantifying the constraint violation degree, improves the expression ability of the multi-dimensional reward by adopting nonlinear reward combination, and introduces the dynamic discount factor to consider short-term and long-term benefits, which significantly enhances the adaptability, optimization effect and continuous performance guarantee of the resource scheduling algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a resource adjustment method, device and equipment of a system. BACKGROUND

[0002] In the field of computing resource scheduling, reinforcement learning method has gradually become a research hotspot due to its adaptability and strong optimization capability. However, current technologies mostly use heuristic rules or relatively simple reinforcement learning strategies for resource allocation, which exposes obvious deficiencies when dealing with complex multi-dimensional optimization objectives. In particular, in the design of reward function, there are three major problems: first, the multi-dimensional reward usually adopts a simple linear combination with fixed weights, which is difficult to reflect the complex relationship and performance trade-off between indicators; second, the hard constraints are handled through threshold limits, which limits the feasible solution space and lacks quantification of the degree of constraint violation, affecting the exploration ability of the algorithm and the quality of the solution; third, too much attention is paid to short-term immediate income, and long-term decision-making and system dynamic changes are ignored, making it difficult to achieve global optimization and continuous performance guarantee. Therefore, the current reward function for resource scheduling has significant defects such as fixed weights, rigid constraint handling, and lack of long-term perspective, which is difficult to meet the optimization needs of complex systems.

[0003] Therefore, how to design a reward function that takes into account multi-dimensional dynamic weights, adaptive constraint violation penalties, and long-term income optimization to improve the decision-making effect of reinforcement learning in complex resource scheduling and the overall performance of the system is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0004] Based on the above problems, the present application provides a resource adjustment method, device and equipment of a system, which can provide a reward function that takes into account multi-dimensional dynamic weights, adaptive constraint violation penalties, and long-term income optimization to improve the decision-making effect of reinforcement learning in complex resource scheduling and the overall performance of the system.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] A resource adjustment method of a system, the method comprising:

[0007] obtaining a current system state and a next system action of a target system; the current system state comprising resource utilization, service quality, system energy consumption and system cost of the target system;

[0008] executing the next system action to obtain a next system state, and calculating a violation penalty in the change process from the current system state to the next system state calculation state in combination with a constraint condition set; the violation penalty being quantitatively calculated based on the degree of constraint violation in the change process from the current system state to the next system state calculation state.

[0009] In combination with the reward dimension mechanism, a reward signal of each reward dimension is calculated according to the current system state and the next system state, so as to obtain a reward signal matrix;

[0010] The reward signal matrix is subjected to nonlinear reward combination to obtain a preliminary reward function, and a dynamic discount factor of the next system state is calculated;

[0011] In combination with the breach penalty and the preliminary reward function, a total reward function is calculated to obtain a total reward function, and a value function is dynamically iteratively updated and calculated in combination with the total reward function and the dynamic discount factor of the next system state, until a converged value function is obtained;

[0012] In combination with the converged value function and the current system state, the target system is subjected to resource adjustment.

[0013] In a possible implementation manner, the total reward function is calculated in combination with the breach penalty and the preliminary reward function, and the value function is dynamically iteratively updated and calculated in combination with the total reward function and the dynamic discount factor of the next system state, until the converged value function is obtained, and the method comprises the following steps.

[0014] The value function is calculated in combination with the total reward function and the dynamic discount factor of the next system state by using a dynamic iterative update calculation formula of the value function;

[0015] If the value function does not converge, the dynamic parameters and hyperparameters in the formula used for iteratively adjusting and calculating the preliminary reward function, the dynamic discount factor, and the value function are adjusted until the converged value function is obtained, and then the adjustment is stopped.

[0016] In a possible implementation manner, the calculation formula of the constraint violation degree is: v j =max (0, g j (s t , a t )), wherein j=1, 2,..., m, m represents a total number of constraints in the constraint condition set; v j represents a constraint violation degree of the jth constraint condition in the constraint condition set; s t represents the current system state; a t represents the next system action; g j (s t , a t ) represents a constraint function value of the corresponding constraint condition, and g j (s t , a t ) is used to measure a state s tThe following action a is performed t The jth constraint condition in the constraint condition set is satisfied j (s t , t ) is less than or equal to 0.

[0017] The calculation formula of the default penalty is: , wherein p j (v j )=-β j ·h(v j );p j (v j ) represents the default penalty of v j ; h(v j ) represents a penalty shape function, h(v j ) includes any one of h(v j )=v j , h(v j )=v j ², h(v j )= -1, h(v j )=log(1+λv j ), λ represents a shape hyperparameter; β j is a dynamic parameter, representing an adaptive penalty coefficient, β j =β j t-1 ·(1+δ·sign(v -ε j )), δ represents an adjustment step, β j t-1 represents the adaptive penalty coefficient in the last iteration update, represents the average value of the constraint violation degree in the last iteration update process; ε j represents a violation threshold.

[0018] In a possible implementation, the nonlinear reward combination of the reward signal matrix to obtain a preliminary reward function includes:

[0019] A correlation matrix of the reward signal matrix is constructed, and each dimension reward in the reward signal matrix is normalized to obtain a normalized reward matrix;

[0020] Each dimension reward in the normalized reward matrix is combined by using a nonlinear combination function combined with the correlation matrix to obtain a preliminary reward function;

[0021] The correlation matrix is: , =1, represents that each reward dimension is completely positively correlated with itself; represents the correlation degree between the i th index and the j th index, ∈[-1,1]; = ;

[0022] The nonlinear combination function is: , R is the preliminary reward function; and represents the reward in the reward signal matrix; represents the cross term function; w i is a dynamic parameter, representing the adaptive weight of r i ; w i =w i （t-1） · (1+η·sign(△Pi)·|△Pi| k ); w i （t -1） represents the weight of the last iteration update of r ; η is a dynamic parameter, representing the learning rate; sign(△Pi) represents a sign function that determines whether to adjust and the adjustment direction according to △Pi; k is a dynamic parameter, representing the adjustment sensitivity parameter; △Pi represents the performance gap, △Pi=(Pi-Ti) / Ti, Pi represents the current system state used when calculating the reward in the reward signal matrix; Ti represents the target state of Pi.

[0023] In a possible implementation, the calculation formula of the dynamic discount factor is: γ(s)=γ base +△γ(s);

[0024] Wherein, γ(s) represents the dynamic discount factor of the system state; γ base is a dynamic parameter, representing the basic discount factor; △γ(s) represents the discount adjustment term of the system state, △γ(s)=ω1·φ1(load(s))+ω2·φ2(scarcity(s))+ω3·φ3(urgency(s)); load(s) represents the system load influencing factor, scarcity(s) represents the resource scarcity influencing factor; urgency(s) represents the task urgency influencing factor; φi(·) is a mapping function that maps each factor to a discount adjustment value; ωi is a dynamic parameter, representing a preset weight coefficient; i takes the value of 1, 2, and 3.

[0025] In a possible implementation, the calculation formula of the total reward function is: R total =R+ ; ​

[0026] Where R is the initial reward function; The penalty for breach of contract is denoted by m, which represents the total number of constraints in the constraint set.

[0027] In one possible implementation, the dynamic iterative update formula for the value function is: Q = Q(s) t a t )+α[R total +γ(s t+1 )·max α Q(s) t+1 ,α)-Q(s t a t )];

[0028] Among them, s t Indicates the current system state; s t+1 Indicates the execution of the next system action a t The next system state obtained thereafter; Q(s) t a t ) indicates the current system state s t Execute the next system action a t The expected cumulative reward value; α is a dynamic parameter representing the learning rate; R total Represents the total reward function; γ(s) t+1 ) represents the dynamic discount factor for the next system state; max α Q(s) t+1 α) represents the next system state s t+1 The maximum value of all possible actions.

[0029] In one possible implementation, the method further includes:

[0030] For the adaptive weight w i Smoothing is performed to obtain smoothed adaptive weights w. i ´, to utilize smooth adaptive weights w i Replace adaptive weights w i To calculate the initial reward function R;

[0031] The smooth adaptive weight w i ´=(1-ρ)·w i +ρ·w i （t-1） ρ represents the smoothing coefficient that controls the stationarity of the weights.

[0032] A system resource adjustment device, the device comprising:

[0033] An acquisition unit is configured to acquire a current system state and a next system action of a target system; the current system state comprises resource utilization, service quality, system energy consumption and system cost of the target system;

[0034] An execution unit is configured to execute the next system action to obtain a next system state;

[0035] A penalty calculation unit is configured to calculate a penalty for breaching a constraint condition set in a process of changing the current system state to the next system state; the penalty for breaching is quantitatively calculated based on a degree of constraint breaching in the process of changing the current system state to the next system state;

[0036] A reward signal matrix calculation unit is configured to calculate a reward signal of each reward dimension according to the current system state and the next system state to obtain a reward signal matrix in combination with a reward dimension mechanism;

[0037] A preliminary reward function calculation unit is configured to perform nonlinear reward combination on the reward signal matrix to obtain a preliminary reward function;

[0038] A dynamic discount factor calculation unit is configured to calculate a dynamic discount factor of the next system state;

[0039] A total reward function calculation unit is configured to calculate a total reward function in combination with the penalty for breaching and the preliminary reward function to obtain the total reward function;

[0040] A value function calculation unit is configured to calculate a total reward function in combination with the penalty for breaching and the preliminary reward function to obtain the total reward function, and to perform dynamic iterative update calculation on a value function in combination with the total reward function and the dynamic discount factor of the next system state until a converged value function is obtained;

[0041] A resource adjustment unit is configured to perform resource adjustment on the target system in combination with the converged value function and the current system state.

[0042] A resource adjustment device of a system, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor; when the processor executes the computer program, a resource adjustment method of the system as claimed in any one of claims 1-8 is implemented.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] This application provides a system resource adjustment method, apparatus, and device. Specifically, when executing the resource adjustment method provided in this application, the current system state (including resource utilization, service quality, system energy consumption, and system cost) and the next system action of the target system are first obtained. This action is executed to obtain the next system state, and the penalty for breach of contract during the change from the current system state to the next system state is calculated based on the constraint set. Next, using a reward dimension mechanism, reward signals for each reward dimension are calculated based on the current system state and the next system state to obtain a reward signal matrix. A nonlinear reward combination is performed on the reward signal matrix to obtain a preliminary reward function, and a dynamic discount factor for the next system state is calculated. Then, the total reward function is calculated by combining the penalty for breach of contract and the preliminary reward function, and the value function is dynamically iteratively updated using the total reward function and the dynamic discount factor to obtain a converged value function. The target system's resources are adjusted based on this value function and the current system state. This application makes constraint handling more flexible by quantitatively calculating the penalty for breach of contract based on the degree of constraint violation. It not only considers the violation of constraints but also accurately quantifies the degree of violation, improving the algorithm's exploration capability and the quality of the solution. Meanwhile, replacing fixed, simple linear combinations with nonlinear reward combinations better reflects the complex relationships and performance trade-offs between different reward dimensions, enhancing the flexibility and adaptability of the reward function. Furthermore, introducing a dynamic discount factor and iteratively updating it effectively balances short-term gains with long-term decisions, ensuring the algorithm can optimize from a global perspective and thus guaranteeing continuous performance. Additionally, adjusting resources for the target system based on the value function and the current system state allows for more effective resource scheduling when the value function converges, improving the system's optimization effect and performance. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a system resource adjustment method provided in this application embodiment;

[0047] Figure 2 This is a schematic diagram of the structure of a resource adjustment device for a system provided in an embodiment of this application. Detailed Implementation

[0048] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0049] In the field of computing resource scheduling, reinforcement learning methods have gradually become a research hotspot due to their adaptability and strong optimization capabilities. However, current technologies often use heuristic rules or relatively simple reinforcement learning strategies for resource allocation. These methods have obvious shortcomings when dealing with complex multi-dimensional optimization objectives. In particular, there are three major problems in the design of reward functions:

[0050] First, multi-dimensional reward signals are usually processed in a simple linear combination manner. The weights of each dimension are fixed and lack a dynamic adjustment mechanism, making it difficult to effectively capture the mutual relationship and cross-influence between different indicators, and thus unable to fully reflect the performance trade-offs in complex optimization objectives.

[0051] Secondly, in the processing of hard constraints, current methods often treat constraint conditions as hard boundaries with threshold limits, resulting in a significant compression of the feasible solution space. Moreover, the degree of constraint violation is not quantitatively evaluated, which not only limits the exploration potential of the algorithm, but also reduces the quality and robustness of the final solution.

[0052] Finally, many methods focus too much on short-term immediate gains, using a fixed discount factor to ignore the long-term impact of decisions and future state predictions, lacking the ability to perceive the dynamic changes in system state and load, and thus making it difficult to achieve global optimization and persistent performance guarantee for the overall system.

[0053] Therefore, the current reward function for resource scheduling has significant defects such as fixed weights, rigid constraint handling, and lack of long-term perspective, making it difficult to meet the optimization needs of complex systems.

[0054] To solve this problem, the embodiment of the present application provides a system resource adjustment method, device and equipment, first, the current state and the next system action of the target system are acquired, the current state covers key indicators such as resource utilization, service quality, system energy consumption and system cost. After executing the next system action, the system enters a new state, and the default constraint condition set is combined to calculate the default penalty in the state change process, and the default penalty is obtained by quantifying the constraint violation degree. Based on the reward dimension mechanism, the dimensional reward signal is calculated according to the current and next system state, and the reward signal matrix is formed. Then, the reward signal matrix is processed by using a nonlinear combination method to obtain a preliminary reward function, and the dynamic discount factor of the next state is calculated. The default penalty and the preliminary reward function are fused to calculate the total reward function, and the value function is iteratively updated combined with the dynamic discount factor until convergence. According to the converged function and the current system state, the target system is adjusted in resource to realize the balance between system performance optimization and constraint satisfaction. The nonlinear reward combination method is adopted in the present application, which can reflect the complex relationship and performance trade-off between indicators compared with the simple fixed weight linear combination. At the same time, the default penalty is calculated by quantifying the constraint violation degree, which can more effectively handle the hard constraint, expand the feasible solution space, and enhance the algorithm exploration ability and solution quality. In addition, the dynamic discount factor and the value function are introduced for updating and iteration, which focuses on long-term decision and system dynamic change, and is helpful to realize global optimization and continuous performance guarantee, and avoids the problem of only focusing on short-term immediate income.

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0056] Referring to Figure 1 , the figure is a method flowchart of a system resource adjustment method provided by the embodiment of the present application, as Figure 1 shown, the system resource adjustment method can include steps S101-S106:

[0057] S101: acquiring the current system state and the next system action of the target system.

[0058] The key of step S101 is to comprehensively and accurately acquire the current state information of the target system and the next system action to be executed, to provide a solid data foundation for subsequent resource adjustment and optimization decision.

[0059] Specifically, the current system state includes four core dimensions: resource utilization, service quality, system energy consumption, and system cost, which comprehensively reflect the system's running status and performance from different aspects.

[0060] Firstly, resource utilization refers to the usage level of hardware resources such as computing resources (such as CPU utilization, memory occupancy), storage resources, or network bandwidth in the system. Efficient resource utilization means that the system can fully exploit the performance potential of existing resources and avoid waste. For example, in a cloud computing platform, a CPU utilization rate of 70% is generally considered ideal, ensuring sufficient processing power while avoiding the risk of overload.

[0061] Secondly, service quality (QoS) reflects the system's response ability and stability to user demand, including response time, task completion rate, reliability, and availability. For example, an online e-commerce platform that can guarantee a 99.9% request success response rate and a page loading time of less than 200 milliseconds indicates a high service quality.

[0062] Thirdly, system energy consumption refers to the total amount of electricity consumed during system operation, which is crucial for energy conservation and emission reduction and reducing operating costs. In data center management, energy consumption can be measured by real-time monitoring of server power consumption indicators, such as a certain computer room with a daily overall energy consumption of 1000 kWh, which can significantly save operating costs by reducing energy consumption by 10%.

[0063] Finally, system cost covers hardware procurement, maintenance, energy consumption, human resource management, and software licensing, etc., which is an important indicator of measuring the economic benefits of the system. For example, an enterprise operating a private cloud platform may have a total monthly cost of around $10,000, including server depreciation fees of $5,000, electricity fees of $2,000, and operation and maintenance personnel salaries of $3,000.

[0064] At the same time, the next system action obtained indicates the scheduling strategy or resource adjustment scheme that the system plans to take, such as increasing the number of virtual machines, adjusting the load balancing strategy, or shutting down idle devices, etc. These actions will directly affect the evolution of the system state. By combining the current state and the next action information, the system can achieve dynamic and adaptive resource management, meeting business needs while considering energy saving and cost control, ultimately achieving the optimal balance between overall performance and economic efficiency.

[0065] S102: Execute the next system action to obtain the next system state, and calculate the violation penalty in the change process of the current system state changing to the next system state calculation state combined with the constraint condition set.

[0066] After performing the next system action, the system will transition from the current state to a new next system state, which reflects the changes in key indicators such as resource allocation, quality of service, energy consumption, and cost. Specifically, the system first adjusts or schedules resources according to the predetermined action instructions, which may include increasing or decreasing the allocation of computing resources, adjusting load balancing strategies, turning on or off part of the equipment, etc., thereby causing changes in the system running environment and performance indicators. After completing the action execution, the specific values and characteristics of the next system state are determined by monitoring and collecting relevant data.

[0067] At the same time, to ensure that the system continuously meets the designed constraints during the adjustment process, the compliance during the state change process needs to be evaluated in combination with the constraint set. The constraint set usually includes resource capacity limits, latency requirements, energy consumption thresholds, budget limits, etc. These constraints aim to ensure the safety, stability, and economy of the system. During the transition from the current system state to the next system state, some constraints may be violated, such as exceeding the maximum capacity of resource usage, exceeding the response time limit, or exceeding the energy consumption limit, etc.

[0068] To quantify the impact of these violations on the overall performance of the system, the application calculates the violation penalty based on the degree of constraint violation. Specifically, instead of simply judging whether the constraint is violated, the severity of the violation is measured through a detailed quantitative method, such as the magnitude and frequency of exceeding the threshold, etc., to more accurately reflect the negative impact of constraint violation on the system. This violation penalty, as a numerical indicator, can be integrated into the subsequent reward function, prompting the algorithm to avoid or reduce constraint violations during optimization, improving the exploration ability of the system and the quality of the solution.

[0069] In summary, by performing the next system action to obtain an updated system state and combining a detailed constraint set, the violation penalty during the state change process is calculated in a quantitative manner, which helps to achieve strict constraint management and scientific optimization of system dynamic adjustment, making resource scheduling both efficient and compliant with multiple complex restrictions, and ensuring the stable and reliable operation of the system.

[0070] In one possible implementation, the calculation formula of the constraint violation degree is: v j =max (0, g j (s t , a t )), where j = 1, 2,..., m, m represents the total number of constraints in the constraint set; v j represents the constraint violation degree of the jth constraint in the constraint set; s t represents the current system state; a tIndicates the next system action; g j (s) t a t ) represents the constraint function value corresponding to the constraint condition, g j (s) t a t ) is used to measure the state s t Next, execute action a t Whether the post-constraint is satisfied by the j-th constraint in the constraint set, g j (s) t a t ) less than or equal to 0;

[0071] The formula for calculating the penalty for breach of contract is as follows: , where p j (v) j )=-β j ·h(v) j ); p j (v) j ) represents v j Penalties for breach of contract; h(v) j ) represents the penalty shape function, h(v) j ) includes h (v j =v j ,h(v j =v j ², h(v) j )= -1, h(v) j ) = log(1 + λv) j Any of the following, where λ represents the shape hyperparameter; β j β is a dynamic parameter representing the adaptive penalty coefficient. j =β j t-1 ·(1+δ·sign( -ε j ), δ represents the adjustment step size, β j t-1 This represents the adaptive penalty coefficient from the previous iteration update. ε represents the average degree of constraint violation during the previous iteration update process; j This indicates a violation of the threshold.

[0072] In one possible implementation, the penalty function is chosen as follows:

[0073] SLA constraints employ exponential penalties: Initial penalty coefficient β1=5.0, shape parameter λ1=2.0; violation threshold ε1=0.05.

[0074] Resource capacity constraint adopts quadratic penalty: p2(v2) = -β 2· v2 2 ; initial penalty coefficient β2 = 10.0; violation threshold ε2 = 0.02.

[0075] Energy consumption constraint adopts linear penalty: p3(v3) = -β 3· v3; initial penalty coefficient β3 = 3.0; violation threshold ε3 = 0.1.

[0076] In one possible implementation, the constraint condition set includes but is not limited to:

[0077] (1) Service Level Agreement (SLA) constraint:

[0078] Common task: response time ≤ 200ms;

[0079] High priority task: response time ≤ 100ms;

[0080] Critical task: response time ≤ 50ms.

[0081] (2) Resource capacity constraint:

[0082] Number of CPU cores ≤ total number of available cores;

[0083] Memory allocation ≤ total amount of available memory;

[0084] Storage allocation ≤ total amount of available storage;

[0085] (3) Energy consumption upper limit constraint:

[0086] Unit time power consumption ≤ upper limit of machine room power;

[0087] Carbon emissions ≤ carbon emission quota.

[0088] In summary, this step ensures that the impact on the system state before performing any system operation can be accurately evaluated, and the potential risks can be controlled and managed by calculating the penalty for violation, so as to realize the efficient and stable operation of the system.

[0089] S103: In combination with the reward dimension mechanism, the reward signals of each reward dimension are calculated according to the current system state and the next system state, and a reward signal matrix is obtained.

[0090] In the system resource adjustment process, the reward dimension mechanism is a key means to achieve multi-objective optimization. This mechanism compares and analyzes the current system state and the next system state obtained after executing the next system action, calculates multiple reward signals in different dimensions, and then forms a structured reward signal matrix, thereby comprehensively reflecting the changes in various aspects of system performance.

[0091] Specifically, system operation involves multiple important indicators such as resource utilization, service quality, system energy consumption, and system cost, etc. These indicators each represent a different aspect of system performance, so optimization of a single indicator often cannot meet complex practical needs. By introducing the reward dimension mechanism, the above multiple indicators can be taken as different reward dimensions, and by calculating the corresponding reward signals for the performance of each dimension, it is helpful for the system to balance various performance goals during resource adjustment and achieve comprehensive optimization.

[0092] The calculation process usually relies on the difference between the current system state and the next system state. For example, in the resource utilization dimension, if the next system state shows that resource usage is more reasonable and overall utilization efficiency is improved, the corresponding reward signal is positive; conversely, if there is a risk of resource waste or overload, the reward signal may be negative. In the service quality dimension, if the response time is shortened, the task success rate is improved, or the system stability is enhanced, positive reward signals will also be generated. The system energy consumption dimension focuses on whether energy consumption is reduced: if the next system state shows that energy consumption is reduced, the reward signal is positive; if energy consumption increases or exceeds a preset threshold, the reward signal is negative.

[0093] The system cost dimension is also very important, as it reflects the impact of resource adjustment on economic costs. If resource adjustment reduces the operating cost of the system, for example, by reducing the use of hardware devices or optimizing software configuration, the corresponding reward signal should be positive; conversely, if resource adjustment leads to an increase in cost, the reward signal may be negative. In this way, the system can comprehensively evaluate the effect of each resource adjustment and make more reasonable and effective decisions accordingly, achieving multi-dimensional optimization and balance.

[0094] In summary, the reward dimension mechanism not only enables the system to evaluate the impact of various operations in detail, but also dynamically adjusts and optimizes system performance during resource adjustment, thereby ensuring that the system meets multiple optimization goals while maintaining efficient and stable operation.

[0095] By way of example, assume there is a cloud computing platform that needs to dynamically allocate computing resources to meet the needs of different services. The system state and actions are as follows:

[0096] System state (current state): Resource utilization: 80%, Quality of service metric (e.g., latency): 50ms, Energy consumption: 1000W, Cost: $20 / hour. Action (decision taken): Increase CPU resources allocated to a certain computing service by 20%.

[0097] After performing the action, the system observes new state information as follows: New resource utilization becomes 85%, new quality of service metric (e.g., latency) becomes 45ms, new energy consumption becomes 950W, and new cost becomes $19 / hour.

[0098] To convert these state changes into reward signals, a reward function needs to be defined for each dimension. Let's assume the following simple reward dimension mechanisms:

[0099] Resource utilization r1: Increase reward if resource utilization increases; otherwise decrease reward.

[0100] Quality of service r2: Increase reward if quality of service improves (e.g., latency decreases); otherwise decrease reward.

[0101] Energy consumption r3: Increase reward if energy consumption decreases; otherwise decrease reward.

[0102] Cost r4: Increase reward if cost decreases; otherwise decrease reward.

[0103] Then the specific calculation process of the reward signal matrix is as follows:

[0104] Resource utilization reward r1: Current utilization increases from 80% to 85%, an increase of 5%. Assume the reward function is 1 point for every 1% increase in resource utilization: r1 = 5 × 1 = 5;

[0105] Quality of service reward r2: Latency decreases from 50ms to 45ms, a decrease of 5ms. Assume a reward of 2 points for every 1ms decrease in latency: r2 = 5 × 2 = 10;

[0106] Energy consumption reward r3: Energy consumption decreases from 1000W to 950W, a decrease of 50W. Assume a reward of 0.1 points for every 1W decrease in energy consumption: r3 = 50 × 0.1 = 5;

[0107] Cost reward r4: Cost decreases from $20 / hour to $19 / hour, a decrease of $1 / hour. Assume a reward of 3 points for every $1 / hour decrease in cost: r4 = 1 × 3 = 3.

[0108] Combine these reward signals into a multi-dimensional raw reward signal (r1, r2, r3, r4):

[0109] (r1, r2, r3, r4) = (5, 10, 5, 3).

[0110] Through the above steps, the state changes and actions of the system can be converted into quantifiable multi-dimensional reward signals. These reward signals are used to evaluate the effects of each action and guide the subsequent decision-making process. This mechanism can help the system continuously optimize resource allocation and achieve a balance and optimization of multi-dimensional goals.

[0111] In one possible implementation, the reward dimension mechanism includes but is not limited to:

[0112] : Resource utilization reward, measuring the utilization efficiency of CPU, memory, etc.:

[0113] : Reward when CPU utilization is within the target interval [60%, 85%];

[0114] : Reward when memory utilization is within the target interval [55%, 80%];

[0115] : Reward when storage utilization is within the target interval [50%, 75%].

[0116] : Service quality reward, measuring response time, throughput, and other service indicators:

[0117] : Reward when response time < 100ms;

[0118] : Reward when throughput reward > 1000 requests / second;

[0119] : Reward when error rate < 0.1%.

[0120] : Energy consumption reward, measuring energy consumption and carbon emissions:

[0121] : Reward when power efficiency > 80% is rewarded;

[0122] : Reward when carbon emissions reward < set threshold.

[0123] : Economic cost reward, measuring operating costs and resource waste:

[0124] : Reward when resource cost < budget limit.

[0125] : Resource waste award <10% when the award is given.

[0126] S104: Nonlinear reward combination is performed on the reward signal matrix to obtain a preliminary reward function, and a dynamic discount factor of the next system state is calculated.

[0127] In the actual system optimization process, simply relying on the independent evaluation of each reward dimension often cannot fully reflect the complex correlation of the overall performance of the system, so it is necessary to nonlinearly combine the reward signal matrix to obtain a more representative comprehensive reward function. Specifically, by introducing a nonlinear combination method, the interaction effect and weight difference between different reward dimensions can be captured, thereby avoiding the information loss or index conflict problem that may be caused by simple weighted summation. For example, when there is a positive or negative correlation between different dimensions, the nonlinear function can appropriately amplify or suppress the influence of certain reward signals, so that the preliminary reward function obtained is more in line with the real needs of the system goal.

[0128] At the same time, the dynamic discount factor γ(s) corresponding to the next system state can be calculated, which is used to measure the value weight of future rewards under the current state. The calculation formula of the dynamic discount factor is.

[0129] γ(s) = γbase + Δγ(s);

[0130] Where γbase represents the base discount factor, which is a dynamic parameter used to ensure that the discount factor fluctuates within a reasonable range; and Δγ(s) is a discount term dynamically adjusted according to the system state characteristics, which is calculated in the following way:

[0131] Δγ(s) = ω1·φ1(load(s)) + ω2·φ2(scarcity(s)) + ω3·φ3(urgency(s)). Here, the system load (load(s)), resource scarcity (scarcity), and task urgency (urgency) are important factors that affect discount adjustment, which are converted to values that can be used for discount adjustment through mapping functions φi(·). Specifically, the mapping function maps these original indicators to a standardized discount adjustment range, ensuring that their contribution to the discount factor is both scientifically based and easy to adjust. The preset weight coefficient ωi is used to control the relative importance of each factor in discount adjustment, and ωi is a dynamic parameter that can achieve flexible adaptation to different operating environments and business scenarios.

[0132] In one possible implementation, the dynamic discount is set as:

[0133] The base discount factor γbase = 0.9;

[0134] System load mapping function: φ1(load) = 0.0.5·tanh(2·load-1); when the load is below 30%, the discount is reduced, more attention is paid to short-term benefits, resources can be allocated more flexibly to improve user experience; when the load is higher than 70%, the discount is increased, more attention is paid to long-term planning, and part of the capacity is reserved to deal with possible future critical tasks.

[0135] Resource scarcity mapping function: φ2(scarcity) = 0.03·scarcity; when the resource is sufficient (scarcity < 0.3), the discount is reduced, when the data center has a large number of idle GPUs, more resources can be allocated to AI training tasks to improve the current training speed; when the resource is scarce (scarcity > 0.7), the discount is increased, when the resource is close to exhaustion, the system needs to be more economical, and priority is given to critical business.

[0136] Task urgency mapping function: φ3(urgency) = -0.04·urgency; when there are many urgent tasks (urgency > 0.7), the discount is reduced, when the system faces a large number of urgent tasks, the current demand should be met first, and the future should not be considered too much; when there are few urgent tasks (urgency < 0.3), the discount is increased, such as night batch processing period, the system can reserve enough resources for long-running training tasks to improve overall throughput.

[0137] In summary, by combining nonlinear reward combination with dynamic discount factor calculation, the preliminary reward function not only accurately reflects the multi-dimensional system performance, but also adjusts the weight of future rewards in real time through the dynamic discount mechanism, balancing the current state and long-term goals, thereby significantly improving the intelligence and robustness of the system resource adjustment strategy.

[0138] In one possible implementation, step S104 performs nonlinear reward combination on the reward signal matrix to obtain a preliminary reward function, including A1-A2:

[0139] A1: Construct a correlation matrix of the reward signal matrix, and normalize each dimension reward in the reward signal matrix to obtain a normalized reward matrix.

[0140] First, according to the correlation between each reward dimension in the reward signal matrix, a correlation matrix is constructed. The diagonal elements of the correlation matrix are all 1, indicating that each reward dimension is completely positively correlated with itself; and the non-diagonal elements reflect the correlation degree between the i-th index and the j-th index, with a value range of [-1, 1], which is used to describe the positive and negative correlation strength and direction between different indexes. At the same time, the rewards in each dimension of the reward signal matrix are normalized to eliminate the differences in the dimensions and numerical ranges of different indexes, thereby obtaining a normalized reward matrix, laying a foundation for subsequent combination calculation.

[0141] wherein the correlation matrix is: .

[0142] =1, which specifies that the elements on the diagonal of the matrix are all equal to 1, indicating that each reward dimension is completely positively correlated with itself. For example, when the CPU utilization rate increases, the CPU utilization rate reward will certainly change, which is a deterministic relationship, ensuring that in the calculation of the total reward, the contribution of each dimension to itself is fully considered.

[0143] represents the degree of correlation between the i-th indicator and the j-th indicator, ∈[-1,1], which limits the value range of the non-diagonal elements and depicts the strength and direction of the correlation between different dimensions. Positive values (such as 0.5): indicate that the two dimensions change in the same direction, for example, when energy consumption increases, economic cost also tends to increase. Negative values (such as -0.7): indicate that the two dimensions change in opposite directions, for example, increasing resource utilization often reduces service quality. Close to zero: indicates that the two dimensions are almost independent and do not affect each other. The value size indicates the correlation strength: |0.9| is stronger than |0.3|.

[0144] = , which ensures the symmetry of the matrix, and the influence of dimension i on dimension j is the same as the influence of dimension j on dimension i. For example, "the influence of CPU usage on memory usage" and "the influence of memory usage on CPU usage" are considered equal when calculating the reward. This symmetry simplifies the mathematical model, ensures the stability of the calculation process, and avoids decision bias.

[0145] In one possible implementation, the correlation matrix can be specifically set as:

[0146]

[0147] This matrix indicates that resource utilization is negatively correlated with service quality (-0.7) and positively correlated with energy consumption (0.5); energy consumption is positively correlated with economic cost (0.6), etc. The construction of the correlation matrix is based on the following data sources: historical running data analysis: correlation analysis is performed on the system running data of the past 6 months; expert experience input: combined with the evaluation results of 5 field experts; online adaptive adjustment: fine-tuning according to real-time feedback during system operation.

[0148] In one possible implementation, the present application adopts an adaptive normalization method to normalize each dimension reward in the reward signal matrix, and the adaptive normalization formula is: .

[0149] wherein μ iand σ i are the mean and standard deviation of dimension i, respectively, dynamically updated by a sliding window method.

[0150] A2: combining each dimension reward in the normalized reward matrix by using a nonlinear combination function based on the correlation matrix to obtain a preliminary reward function.

[0151] Based on the normalized reward matrix and the correlation matrix, each dimension reward is organically fused by using a nonlinear combination function to form a preliminary reward function. The preliminary reward function not only contains linear weighted items of each reward dimension, but also introduces cross-item functions to capture the mutual influence and synergistic effect between different dimensions.

[0152] In a possible implementation manner, the nonlinear combination function is: , R is the preliminary reward function; and represents the reward in the reward signal matrix; represents a cross-item function; w i is a dynamic parameter, representing the adaptive weight of r i .

[0153] It should be noted that the weight w i of each reward dimension is a dynamic adaptive parameter, which is continuously adjusted according to system performance feedback. The weight update formula is: w i =w i （t-1） · (1+η·sign (△Pi) · |△Pi| k ).

[0154] wherein w i （t-1） represents the weight value at the last iteration; η is a dynamic parameter, which is a learning rate control parameter and determines the weight adjustment amplitude; sign (△Pi) is a sign function, which is used to indicate the adjustment direction (increase or decrease); k is a dynamic sensitivity parameter, which adjusts the nonlinearity degree of adjustment; the performance gap ΔPi is:△Pi= (Pi-Ti) / Ti, wherein Pi is the actual performance value of the corresponding reward dimension under the current system state, and Ti is the target performance value of the index. Through the mechanism, the weight can be intelligently corrected according to the deviation between the actual and the target, so that the combined reward function can more accurately reflect the current performance and optimization demand of the system.

[0155] In a possible implementation manner, the initial weight is set as: w1=0.3, w2=0.3, w3=0.2, w4=0.2; the learning rate is: η=0.05; and the sensitivity parameter is: k=0.5.

[0156] It should be further noted that, It is a cross-term function used to model dimensions. and dimensions The interaction effect. The design of the cross-term function considers the following forms:

[0157] Product form: ;

[0158] Exponential form: ;

[0159] Threshold form: ,

[0160] Where α is the cross strength parameter, θ i and θ j It is a threshold parameter.

[0161] In one possible implementation, to ensure the weight sum is 1, it is necessary to apply the weights to any r. i Adaptive weights w i Normalization is performed: .

[0162] In one possible implementation, to avoid drastic fluctuations in the weights, the adaptive weights w can be adjusted. i Smoothing is performed to obtain smoothed adaptive weights w. i ´, to utilize smooth adaptive weights w i Replace adaptive weights w i To calculate the initial reward function R. i ´=(1-ρ)·w i +ρ·w i （t-1） ρ represents the smoothing coefficient that controls the stationarity of the weights, and the smoothing coefficient ρ = 0.8.

[0163] S105: Calculate the total reward function by combining the default penalty and the preliminary reward function, and then calculate the total reward function by combining the default penalty and the preliminary reward function. Finally, dynamically iterate and update the value function by combining the total reward function and the dynamic discount factor of the next system state until a convergent value function is obtained.

[0164] By combining the penalty for breach of contract and the initial reward function, the total reward function of the system can be calculated. Specifically, the total reward function R... total It consists of the initial reward function R and the sum of the penalty terms for default under all constraints, and its calculation formula is as follows:

[0165] R total =R+ ;

[0166] Where R is the initial reward function; The penalty for breach of contract is defined as m, which represents the total number of constraints in the constraint set. This design ensures that while rewarding system performance, it effectively considers and punishes system violations of constraints, thereby promoting the resource adjustment process to meet relevant restrictions.

[0167] Based on this total reward function, and the dynamic discount factor γ(s) corresponding to the next system state t+1 The strategy can be continuously optimized by dynamically iteratively updating the value function Q. The update formula is:

[0168] Q = Q(s) t a t )+α[R total +γ(s t+1 )·max α Q(s) t+1 ,α)-Q(s t a t )]

[0169] Among them, s t Indicates the current system state; s t+1 Indicates the execution of the next system action a t The next system state obtained thereafter; Q(s) t a t ) indicates the current system state s t The expected cumulative reward value for executing the next system action at; α is a dynamic parameter representing the learning rate; R total Represents the total reward function; γ(s) t+1 ) represents the dynamic discount factor for the next system state; max α Q(s) t+1 α) represents the next system state s t+1 The maximum value of all possible actions.

[0170] Through the iterative update process described above, the value function is gradually adjusted, enabling the system to more accurately assess the long-term benefits and risk trade-offs of different actions. When the value function converges, it indicates that the strategy has stabilized. At this point, effective resource adjustments can be made to the target system based on the value function to achieve comprehensive optimization of system performance and constraints.

[0171] In one possible implementation, step S105 involves calculating the total reward function by combining the default penalty and the initial reward function, and then dynamically iteratively updating the value function by combining the total reward function and the dynamic discount factor of the next system state until a convergent value function is obtained, including:

[0172] The value function is calculated by using a dynamic iterative updating calculation formula, in combination with the total reward function and a dynamic discount factor of the next system state. If the value function does not converge, the dynamic parameters and hyperparameters in the formula used for calculating the preliminary reward function, the dynamic discount factor and the value function are iteratively adjusted until the convergent value function is obtained, and then the iteration is stopped.

[0173] Specifically, first, the value function is calculated by using the current total reward function and the dynamic discount factor of the next system state according to the dynamic iterative updating calculation formula, to obtain the latest value function. Then, it is determined whether the value function has converged. If the convergence condition is not met, the system adjusts and optimizes various dynamic parameters and hyperparameters used in the calculation of the preliminary reward function, the dynamic discount factor and the value function, to obtain a convergent value function by using the dynamic iterative updating calculation formula in combination with a new total reward function and a new dynamic discount factor of the next system state.

[0174] This process is repeatedly executed until the latest value function meets the convergence requirement, and the iteration is stopped. Through the closed-loop adaptive iteration mechanism, the stability and accuracy of the value function can be effectively improved, thereby providing a reliable foundation for subsequent resource adjustment strategies and improving the overall performance and constraint satisfaction of the system.

[0175] S106: Adjusting resources of the target system in combination with the convergent value function and the current system state.

[0176] After obtaining the value function, resources of the target system can be adjusted in combination with the value function, so as to select the optimal resource allocation strategy by using the state-action value evaluation reflected by the function. Through the trade-off and prediction of future rewards by the value function, the key elements such as computing resources, storage resources and network bandwidth of the target system can be reasonably adjusted to improve overall performance indicators such as response speed, energy efficiency and service quality. At the same time, the adjustment process comprehensively considers the current system state and constraint conditions to ensure the effectiveness and stability of resource use, thereby realizing dynamic optimization and adaptive management of resource allocation and promoting efficient operation of the target system in a complex environment.

[0177] In a possible implementation manner, the adjusting resources of the target system in combination with the convergent value function and the current system state comprises:

[0178] According to the convergent value function and the current system state, a state-action value is output, and resources of the target system are adjusted in combination with the state-action value.

[0179] The specific process includes: first, inputting the current system state into the converged value function, the value function calculating and outputting the state-action value corresponding to the corresponding next time executable action under the state. Subsequently, based on the state-action value, the target system is adjusted in resource, realizing the dynamic optimization of resource allocation. In this way, the resource adjustment strategy not only makes full use of the accurate estimation of future income of the value function, but also ensures that each adjustment tends to improve the overall system performance and benefit, with strong real-time adaptability and optimization effect.

[0180] Based on the content of S101-S106, it can be known that the current system state and the predicted next system action of the target system are collected first. The current system state covers key indicators such as resource utilization, service quality, system energy consumption and system cost. After the predicted next system action is executed, a new system state is obtained, and at the same time, any constraint violation occurring in the transition process from the current state to the next state is quantified according to a set of constraint conditions to calculate the penalty for violation. The violation penalty is obtained based on quantitative analysis of the degree of violation. Next, through the reward dimension mechanism, the reward signal of each dimension is calculated according to the current system state and the next system state, and finally a reward signal matrix is formed. Then, a preliminary reward function is generated by applying a nonlinear reward combination strategy to the reward signal matrix, and a dynamic discount factor of the next system state is calculated. On this basis, the total reward function is calculated in combination with the violation penalty and the preliminary reward function, and then the value function is dynamically iteratively updated in combination with the total reward function and the dynamic discount factor, until the value function converges. When this convergence state is reached, the value function is used to implement resource adjustment on the target system to achieve optimal resource allocation and efficiency improvement. The present application quantitatively calculates the violation penalty based on the degree of constraint violation, making the constraint processing more flexible, not only considering the violation of the constraint, but also accurately quantifying the degree of violation, improving the exploration ability of the algorithm and the quality of the solution. At the same time, the nonlinear reward combination is used instead of the fixed simple linear combination, which better reflects the complex relationship and performance trade-off between different reward dimensions, and improves the flexibility and adaptability of the reward function. In addition, the dynamic discount factor is introduced and iteratively updated, effectively balancing short-term income and long-term decision-making, ensuring that the algorithm can optimize from a global perspective and achieve continuous performance guarantee.

[0181] Reference Figure 2 , Figure 2 A structural schematic diagram of a resource adjustment device of a system provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the resource adjustment device of the system includes: Figure 2

[0182] ​The acquisition unit 201 is configured to acquire a current system state and a next system action of a target system; the current system state comprises resource utilization, service quality, system energy consumption and system cost of the target system;

[0183] The execution unit 202 is configured to execute the next system action to obtain a next system state;

[0184] The penalty calculation unit 203 is configured to calculate a penalty for violation in a process of changing the current system state to a calculated state of the next system state in combination with a constraint condition set; the penalty for violation is quantitatively calculated based on a constraint violation degree in the process of changing the current system state to the calculated state of the next system state;

[0185] The reward signal matrix calculation unit 204 is configured to calculate a reward signal of each reward dimension according to the current system state and the next system state in combination with a reward dimension mechanism to obtain a reward signal matrix;

[0186] The preliminary reward function calculation unit 205 is configured to perform nonlinear reward combination on the reward signal matrix to obtain a preliminary reward function;

[0187] The dynamic discount factor calculation unit 206 is configured to calculate a dynamic discount factor of the next system state;

[0188] The total reward function calculation unit 207 is configured to perform total reward function calculation in combination with the penalty for violation and the preliminary reward function to obtain a total reward function;

[0189] The value function calculation unit 208 is configured to perform total reward function calculation in combination with the penalty for violation and the preliminary reward function to obtain a total reward function, and perform dynamic iterative update calculation on a value function in combination with the total reward function and the dynamic discount factor of the next system state, until a converged value function is obtained;

[0190] The resource adjustment unit 209 is configured to perform resource adjustment on the target system in combination with the converged value function and the current system state.

[0191] In a possible implementation, the value function calculation unit 208 is specifically configured to:

[0192] The value function is calculated by using a dynamic iterative update calculation formula of the value function in combination with the total reward function and the dynamic discount factor of the next system state;

[0193] If the value function does not converge, the dynamic parameters and hyperparameters in the formula used for iteratively adjusting and calculating the preliminary reward function, the dynamic discount factor and the value function are adjusted until the converged value function is obtained, and then the adjustment is stopped.

[0194] In a possible implementation, the calculation formula of the constraint violation degree is: v j =max (0, g j (s t , a t )), where j = 1, 2,..., m, m represents the total number of constraints in the constraint set; v j represents the constraint violation degree of the jth constraint in the constraint set; s t represents the current system state; a t represents the next system action; g j (s t , a t ) represents the constraint function value of the corresponding constraint, g j (s t , a t ) is used to measure whether the constraint is satisfied after the action a t is performed in the state s t , g j (s t , a t ) is less than or equal to 0;

[0195] The calculation formula of the default penalty is: where p j (v j ) = -β j · h (v j ) ; p j (v j ) represents the default penalty of v j ; h (v j ) represents a penalty shape function, h (v j ) includes any one of h (v j ) = v j , h (v j ) = v j ², h (v j ) = |v j | - 1, h (v j ) = log (1 + λv j ), λ represents a shape hyperparameter; β j is a dynamic parameter, representing an adaptive penalty coefficient, β j = β t-1 j · (1 + δ · sign (v - ε j ) ), δ represents an adjustment step, β t-1 j represents the adaptive penalty coefficient when the last iteration is updated, ε represents the average degree of constraint violation during the previous iteration update process; j This indicates a violation of the threshold.

[0196] In one possible implementation, the step of performing a nonlinear reward combination on the reward signal matrix to obtain a preliminary reward function includes:

[0197] Construct the correlation matrix of the reward signal matrix, and normalize the rewards of each dimension in the reward signal matrix to obtain the normalized reward matrix;

[0198] By combining the correlation matrix with a nonlinear combination function, the rewards of each dimension in the normalized reward matrix are combined to obtain a preliminary reward function;

[0199] The correlation matrix is ​​as follows: , =1, This indicates that each reward dimension is perfectly positively correlated with itself; This indicates the degree of correlation between the i-th indicator and the j-th indicator. ∈[-1,1]; = ;

[0200] The nonlinear combination function is: R is the initial reward function; and This represents the reward in the reward signal matrix; Represents the cross term function; w i For dynamic parameters, r represents i Adaptive weights; w i =w i （t-1） ·(1+η·sign(△Pi)·|△Pi| k ); wi （t -1） Indicates the last iteration update The weights are: η is a dynamic parameter representing the learning rate; sign(ΔPi) is a sign function that determines whether to adjust and the direction of adjustment based on ΔPi; k is a dynamic parameter representing the adjustment sensitivity parameter; ΔPi represents the performance gap, ΔPi = (Pi - Ti) / Ti, where Pi represents the reward in the calculated reward signal matrix. The current system state used at that time; Ti represents the target state of Pi.

[0201] In one possible implementation, the dynamic discount factor is calculated as: γ(s) = γ base +△γ(s;

[0202] Where γ(s) represents the dynamic discount factor of the system state; γ base φi is a dynamic parameter representing the basic discount factor; Δγ(s) represents the discount adjustment term for the system state, Δγ(s) = ω1·φ1(load(s)) + ω2·φ2(scarcity(s)) + ω3·φ3(urgency(s)); load(s) represents the system load influencing factor, scarcity(s) represents the resource scarcity influencing factor, urgency(s) represents the task urgency influencing factor; φi(·) is a mapping function that maps each factor to the discount adjustment value; ωi is a dynamic parameter representing the preset weight coefficient; the value of i is 1, 2, or 3.

[0203] In one possible implementation, the total reward function is calculated as follows: R total =R+ ;

[0204] Where R is the initial reward function; The penalty for breach of contract is denoted by m, which represents the total number of constraints in the constraint set.

[0205] In one possible implementation, the dynamic iterative update formula for the value function is: Q = Q(s) t a t )+α[R total +γ(s t+1 )·max α Q(s) t+1 ,α)-Q(s t a t )];

[0206] Among them, s t Indicates the current system state; s t+1 Indicates the execution of the next system action a t The next system state obtained thereafter; Q(s) t a t ) indicates the current system state s t The expected cumulative reward value for executing the next system action at; α is a dynamic parameter representing the learning rate; R total Represents the total reward function; γ(s) t+1 ) represents the dynamic discount factor for the next system state; max α Q(s) t+1 α) represents the next system state s t+1 The maximum value of all possible actions.

[0207] In one possible implementation, the device further includes:

[0208] a smoothing unit configured to smooth the adaptive weight w i to obtain a smoothed adaptive weight w i ´, so as to replace the adaptive weight w i with the smoothed adaptive weight w i to calculate the preliminary reward function R;

[0209] the smoothed adaptive weight w i ´= (1-ρ)·w i + ρ·w i （t-1） , wherein ρ represents a smoothing coefficient for controlling weight stability.

[0210] In addition, the embodiments of the present application further provide a system resource adjustment device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the system resource adjustment method is realized.

[0211] The present application realizes flexible adjustment of constraint processing by quantitatively calculating default penalty based on constraint violation degree, considers whether the constraint is violated and accurately measures the severity of the violation, thereby improving the exploration ability of the algorithm and the quality of the solution. At the same time, the nonlinear reward combination is used instead of the fixed linear weighting, which more truly reflects the complex relationship and performance trade-off between each reward dimension, enhances the expression ability and adaptability of the reward function. Furthermore, the dynamic discount factor is introduced and updated through iteration, effectively balancing short-term income and long-term decision-making, so that the algorithm can optimize resource scheduling from a global perspective and guarantee the sustained and stable performance of the system.

[0212] The above describes in detail the system resource adjustment method, device and equipment provided by the present application. The embodiments in the specification are described in a progressive manner, and each embodiment mainly describes the differences from other embodiments. The same or similar parts of each embodiment can be referred to. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0213] It should be understood that, in the present application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three kinds of relationships, for example, "A and / or B" can represent: only A, only B and A and B exist at the same time Three cases, where A, B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c, can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

Claims

1. A method of resource adjustment of a system, characterized by, The method comprises: obtaining a current system state and a next system action of a target system; the current system state comprises resource utilization, service quality, system energy consumption and system cost of the target system; performing the next system action to obtain a next system state, and calculating a violation penalty in a change process of the current system state to a calculated state of the next system state in combination with a constraint condition set; the violation penalty is quantitatively calculated based on a constraint violation degree in the change process of the current system state to the calculated state of the next system state; combining a reward dimension mechanism, calculating a reward signal of each reward dimension according to the current system state and the next system state to obtain a reward signal matrix; performing nonlinear reward combination on the reward signal matrix to obtain a preliminary reward function, and calculating a dynamic discount factor of the next system state; combining the violation penalty and the preliminary reward function to calculate a total reward function, and performing dynamic iterative update calculation on a value function in combination with the total reward function and the dynamic discount factor of the next system state until a converged value function is obtained; combining the converged value function and the current system state to perform resource adjustment on the target system; The calculation formula of the dynamic discount factor is: γ(s) = γ base +△γ(s); wherein γ(s) represents a dynamic discount factor of the system state; γ base is a dynamic parameter, representing a basic discount factor; Δγ(s) represents a discount adjustment term of the system state, Δγ(s) = ω1·φ1(load(s))+ω2·φ2(scarcity(s))+ω3·φ3(urgency(s)); load(s) represents a system load influencing factor, scarcity(s) represents a resource scarcity influencing factor; urgency(s) represents a task urgency influencing factor; φi(·) is a mapping function, mapping each factor to a discount adjustment value; ωi is a dynamic parameter, representing a preset weight coefficient; i takes values of 1, 2, and 3; the nonlinear reward combination on the reward signal matrix to obtain the preliminary reward function comprises: constructing a correlation matrix of the reward signal matrix, and performing normalization on each dimension reward in the reward signal matrix to obtain a normalized reward matrix; combining each dimension reward in the normalized reward matrix by using a nonlinear combination function in combination with the correlation matrix to obtain the preliminary reward function; Wherein, the correlation matrix is: , =1, indicates that each reward dimension is completely positively correlated with itself; indicates the correlation degree between the i th index and the j th index, ∈[-1,1]; = ; The nonlinear combination function is: R is the initial reward function; and This represents the reward in the reward signal matrix; Represents the cross term function; w i For dynamic parameters, r represents i Adaptive weights; w i =w i （t-1） ·(1+η·sign(△Pi)·|△Pi| k ); w i （t-1） Indicates the last iteration update The weights are: η is a dynamic parameter representing the learning rate; sign(ΔPi) is a sign function that determines whether to adjust and the direction of adjustment based on ΔPi; k is a dynamic parameter representing the adjustment sensitivity parameter; ΔPi represents the performance gap, ΔPi = (Pi - Ti) / Ti, where Pi represents the reward in the calculated reward signal matrix. The current system state used at that time; Ti represents the target state of Pi.

2. The method of claim 1, wherein, the combination of the violation penalty and the preliminary reward function to calculate the total reward function, and the dynamic iterative update calculation on the value function in combination with the total reward function and the dynamic discount factor of the next system state until the converged value function is obtained, comprises: calculating the value function in combination with the total reward function and the dynamic discount factor of the next system state by using a dynamic iterative update calculation formula of the value function; if the value function does not converge, iteratively adjusting dynamic parameters and hyperparameters in formulas used in the calculation of the preliminary reward function, the dynamic discount factor and the value function until the converged value function is obtained, and then stopping.

3. The method of claim 1, wherein, The calculation formula of the constraint violation degree is: v j =max (0, g j (s t , a t )), wherein j=1, 2,..., m, m represents the total number of constraints in the constraint set; v j represents the constraint violation degree of the jth constraint in the constraint set; s t represents the current system state; a t represents the next system action; g j (s t , a t ) represents the constraint function value of the corresponding constraint condition, g j (s t , a t ) is used to measure whether the constraint is satisfied after the action a t is performed in the state s t , g j (s t , a t ) is less than or equal to 0; The calculation formula of the penalty is: where p j (v j )=-β j ·h(v j );p j (v j ) represents the violation penalty of v j ; h(v j ) represents a penalty shape function, h(v j ) includes any one of h(v j )=v j , h(v j )=v j ², h(v j )= -1, h(v j )=log(1+λv j ), and λ represents a shape hyperparameter; β j is a dynamic parameter, representing an adaptive penalty coefficient, β j =β j t-1 ·(1+δ·sign(v -ε j )), δ represents an adjustment step, β j t-1 represents the adaptive penalty coefficient in the last iteration update, represents the average value of the constraint violation degree in the last iteration update process; ε j represents a violation threshold.

4. The method of claim 1, wherein, The total reward function is calculated as follows: R total = R + R ; where R is the preliminary reward function; For the breach penalty, m represents the total number of constraints in the constraint set.

5. The method according to claim 1 or 2, characterized in that, The formula for the dynamic iterative update of the value function is: Q = Q(s) t a t )+α[R total +γ(s t+1 )·max α Q(s) t+1 ,α)-Q(s t a t )]; where s t represents the current system state; s t+1 represents the next system state after performing the next system action a t ; Q(s t , a t ) represents the expected cumulative reward value of performing the next system action a t at the current system state s t ; a is a dynamic parameter representing a learning rate; R total represents the total reward function; g(s t+1 ) represents a dynamic discount factor of the next system state; and max α Q(s t+1 , a) represents the maximum value of all possible actions in the next system state s t+1 .

6. The method of claim 1, wherein, The method further comprises: The adaptive weight w i is smoothed to obtain a smoothed adaptive weight w i ´, so that the preliminary reward function R is calculated by using the smoothed adaptive weight w i ´ instead of the adaptive weight w i . The smoothing adaptive weight w i ´= (1 - p) - w i + p - w i （t-1） p represents a smoothing coefficient for controlling the smoothness of the weight.

7. A system resource adjustment apparatus, characterized by comprising: The device comprises: an acquisition unit configured to obtain a current system state and a next system action of a target system; the current system state comprises resource utilization, service quality, system energy consumption and system cost of the target system; an execution unit configured to perform the next system action to obtain a next system state; a violation penalty calculation unit configured to calculate a violation penalty in a change process of the current system state to a calculated state of the next system state in combination with a constraint condition set; the violation penalty is quantitatively calculated based on a constraint violation degree in the change process of the current system state to the calculated state of the next system state; a reward signal matrix calculation unit, configured to calculate a reward signal of each reward dimension according to the current system state and the next system state in combination with a reward dimension mechanism, to obtain a reward signal matrix; a preliminary reward function calculation unit, configured to perform nonlinear reward combination on the reward signal matrix to obtain a preliminary reward function; a dynamic discount factor calculation unit, configured to calculate a dynamic discount factor of the next system state; a total reward function calculation unit, configured to perform total reward function calculation on the preliminary reward function in combination with the default penalty to obtain a total reward function; a value function calculation unit, configured to perform total reward function calculation on the preliminary reward function in combination with the default penalty to obtain a total reward function, and to perform dynamic iterative update calculation on a value function in combination with the total reward function and the dynamic discount factor of the next system state until a converged value function is obtained; a resource adjustment unit, configured to perform resource adjustment on the target system in combination with the converged value function and the current system state; The calculation formula of the dynamic discount factor is: γ(s) = γ base +△γ(s); wherein γ(s) represents a dynamic discount factor of the system state; γ base is a dynamic parameter, representing a basic discount factor; Δγ(s) represents a discount adjustment term of the system state, Δγ(s) = ω1·φ1(load(s))+ω2·φ2(scarcity(s))+ω3·φ3(urgency(s)); load(s) represents a system load influencing factor, scarcity(s) represents a resource scarcity influencing factor; urgency(s) represents a task urgency influencing factor; φi(·) is a mapping function, mapping each factor to a discount adjustment value; ωi is a dynamic parameter, representing a preset weight coefficient; i takes values of 1, 2, and 3; the preliminary reward function obtained by performing nonlinear reward combination on the reward signal matrix comprises: constructing a correlation matrix of the reward signal matrix, and performing normalization on each dimension reward in the reward signal matrix to obtain a normalized reward matrix; combining each dimension reward in the normalized reward matrix by using a nonlinear combination function in combination with the correlation matrix to obtain the preliminary reward function; Wherein, the correlation matrix is: , =1, indicates that each reward dimension is completely positively correlated with itself; indicates the correlation degree between the i th index and the j th index, ∈[-1,1]; = ; The nonlinear combination function is: R is the initial reward function; and This represents the reward in the reward signal matrix; Represents the cross term function; w i For dynamic parameters, r represents i Adaptive weights; w i =w i （t-1） ·(1+η·sign(△Pi)·|△Pi| k ); w i （t-1） Indicates the last iteration update The weights are: η is a dynamic parameter representing the learning rate; sign(ΔPi) is a sign function that determines whether to adjust and the direction of adjustment based on ΔPi; k is a dynamic parameter representing the adjustment sensitivity parameter; ΔPi represents the performance gap, ΔPi = (Pi - Ti) / Ti, where Pi represents the reward in the calculated reward signal matrix. The current system state used at that time; Ti represents the target state of Pi.

8. A system resource adjustment device, characterized by comprising: comprise: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, a resource adjustment method of the system as claimed in any one of claims 1-6 is implemented.

Images