Power service system tuning method, device and medium for autonomous controllable environment

Abstract

The application discloses a power service system optimization method and device for an autonomous controllable environment and a medium, comprising: acquiring a service load fluctuation rate and a resource health index of a power service system; determining a target strategy model according to the service load fluctuation rate and the resource health index, the target strategy model comprising at least one of an offline strategy model, an online strategy model and a preset rule model; acquiring target feature data of the power service system, inputting the target feature data into the target strategy model and obtaining a target resource scheduling strategy of the power service system. The application solves the problems of staticity, insufficient flexibility and weak uncertainty processing capability of the existing power service system management mode, and realizes the optimization of the scheduling strategy of the power service system resource. The power service system for autonomous controllable technology is in the initial stage of construction, the application uses the method of artificial intelligence, finally realizes the explainable performance optimization, and improves the performance index of the business system after the signal creation adaptation reconstruction.

Description

Technical Field

[0001] This invention relates to the field of power business system optimization technology, and in particular to power business system optimization methods, equipment and media for autonomous and controllable environments. Background Technology

[0002] Currently, the management of power business systems largely relies on traditional manual or semi-automated management methods. Critical business support systems such as database systems, task scheduling platforms, and business computing engines require manual intervention for configuration in actual operation, resulting in low resource allocation efficiency. This problem is particularly pronounced in power business system environments that are independently controllable and in their early stages of development, where the lack of long-term operational logs and mature prior knowledge bases exacerbates the complexity of the architecture and the high management requirements.

[0003] While manual management can handle complex situations based on experience, it is prone to errors and inefficient when dealing with large volumes of data and frequent real-time changes in power business systems. It struggles to handle large volumes of tasks and complex real-time data in a short period. In power business processing, traditional information systems lack intelligent optimization capabilities to adapt to large-scale data flows, dynamic workloads, and peak-valley fluctuations in system resources, making it difficult to achieve adaptive matching of computing resources and workloads. Some power business systems employ rule-based scheduling or decision support systems, which can automate task scheduling to some extent by manually inputting large amounts of historical and real-time data and using models to predict task requirements and database status. However, these systems typically rely on static rules and pre-set models, lacking the ability to adapt to real-time data changes, and are unable to effectively handle uncertainties in power business systems or make dynamic strategy adjustments. Summary of the Invention

[0004] This invention provides a method, equipment, and medium for optimizing power business systems in an autonomous and controllable environment, in order to optimize the scheduling strategy of power business system resources, improve the automation level of power business system scheduling management, and thereby improve resource utilization, task response speed, system stability, and overall service quality.

[0005] According to one aspect of the present invention, a method for optimizing a power business system for an autonomous and controllable environment is provided, comprising:

[0006] Obtain the load volatility and resource health index of the power business system;

[0007] Based on the business load volatility and resource health index, a target strategy model is determined, wherein the target strategy model includes at least one of the following: offline strategy model, online strategy model, and preset rule model.

[0008] Obtain target feature data from the power business system, input the target feature data into the target strategy model, and obtain the target resource scheduling strategy of the power business system.

[0009] The target feature data includes at least one of the following: service load features, resource usage features, configuration status features, fault and anomaly features, and environmental disturbance features.

[0010] In some embodiments, a target strategy model is determined based on business load volatility and resource health index, including:

[0011] If the business load volatility is less than the first preset volatility and the resource health index is greater than the first health index, the offline strategy model will be determined as the target strategy model.

[0012] If the volatility of the business load is greater than the first preset volatility but less than the second preset volatility, the online strategy model will be determined as the target strategy model.

[0013] If the business load volatility is greater than the second preset volatility or the resource health index is less than the second health index, the preset rule model will be determined as the target strategy model.

[0014] In some embodiments, after obtaining the target resource scheduling strategy of the power business system, the method further includes:

[0015] Determine the scenario scoring weights for the power business system and the current resource scheduling strategy of the power system;

[0016] The current resource scheduling strategy and the target resource scheduling strategy are weighted based on the scenario scoring weights to obtain the weighted resource scheduling strategy.

[0017] Based on a weighted resource scheduling strategy, the system resources of the power business system are scheduled and allocated. The system resources include at least one of CPU, I / O, memory and network bandwidth.

[0018] In some embodiments, the training process of the offline policy model includes:

[0019] Construct a system environment model corresponding to the power business system, which includes a state space, action space, and reward function;

[0020] Based on system environment modeling, construct the initial policy network and the initial value network;

[0021] Obtain historical log data corresponding to the power business system, generate interactive data corresponding to the historical log data, and store the interactive data in the experience pool;

[0022] Batch sampling data is obtained from the experience pool, and the network parameters of the initial policy network and the initial value network are updated based on the batch sampling data and the near-end policy optimization algorithm.

[0023] When the initial policy network meets the preset conditions, the initial policy network that meets the preset conditions is determined as the offline policy model.

[0024] In some embodiments, updating the network parameters of the initial policy network based on batch sampling data and a near-end policy optimization algorithm includes:

[0025] Based on the batch sampling data, the value of the first advantage function is calculated through the first advantage function; wherein, the first advantage function is determined based on the cumulative discount reward and state value function corresponding to the batch sampling data, and the state value function is determined based on the initial value network;

[0026] Determine the probability ratio of the new and old policies corresponding to the initial policy network, and the shearing threshold corresponding to the near-end policy optimization algorithm;

[0027] Based on the first advantage function value, the probability ratio of the new and old policies, and the pruning threshold, the pruning objective function is maximized to update the network parameters of the initial policy network, thus obtaining the offline policy model.

[0028] In some embodiments, it also includes:

[0029] Based on state value and cumulative discounted reward, a mean squared error function is determined and minimized to update the network parameters of the initial value network, thus obtaining an offline value model.

[0030] In some embodiments, the training process of the online policy model includes:

[0031] Based on the Prometheus monitoring system and cloud management platform, real-time feature data of the power business system is obtained, and the real-time feature data is input into the offline strategy model to obtain the action probability distribution.

[0032] Based on the action probability distribution, the target action value is determined, and the target state value is determined through evaluation using a target value network.

[0033] The second advantage function is calculated based on the target state value and the target action value, and the policy loss function is constructed based on the second advantage function, the policy entropy, and the above action probability distribution.

[0034] A value loss function is constructed, and the offline policy model and offline value model are optimized based on the policy loss function and value loss function, respectively, to obtain the online policy model.

[0035] In some embodiments, optimizing the offline policy model and the offline value model based on the policy loss function and the value loss function, respectively, includes:

[0036] The offline strategy model and / or offline value model are identified as the models to be optimized, and the current network parameters and target network parameters of the models to be optimized are determined.

[0037] The current network parameters and the target network parameters are weighted based on a preset update speed to obtain the network parameters to be updated.

[0038] Based on the network parameters to be updated, the current network parameters are gradually updated to the target network parameters.

[0039] According to another aspect of the present invention, a power service system optimization device for an autonomous and controllable environment is provided, comprising:

[0040] The load and health index determination module is used to obtain the business load volatility and resource health index of the power business system;

[0041] The target strategy model determination module is used to determine the target strategy model based on the business load volatility and resource health index. The target strategy model includes at least one of the following: offline strategy model, online strategy model, and preset rule model.

[0042] The resource scheduling strategy generation module is used to acquire target feature data of the power business system, input the target feature data into the target strategy model, and obtain the target resource scheduling strategy of the power business system.

[0043] The target feature data includes at least one of the following: service load features, resource usage features, configuration status features, fault and anomaly features, and environmental disturbance features.

[0044] Optional, the target strategy model determination module, specifically used for:

[0045] If the business load volatility is less than the first preset volatility and the resource health index is greater than the first health index, the offline strategy model will be determined as the target strategy model.

[0046] If the volatility of the business load is greater than the first preset volatility but less than the second preset volatility, the online strategy model will be determined as the target strategy model.

[0047] If the business load volatility is greater than the second preset volatility or the resource health index is less than the second health index, the preset rule model will be determined as the target strategy model.

[0048] Optionally, the power business system optimization device for an autonomous and controllable environment also includes a dispatch module, specifically used for:

[0049] After obtaining the target resource scheduling strategy of the power business system, determine the scenario scoring weight corresponding to the power business system, as well as the current resource scheduling strategy of the power system;

[0050] The current resource scheduling strategy and the target resource scheduling strategy are weighted based on the scenario scoring weights to obtain the weighted resource scheduling strategy.

[0051] Based on a weighted resource scheduling strategy, the system resources of the power business system are scheduled and allocated. The system resources include at least one of CPU, I / O, memory and network bandwidth.

[0052] Optionally, the power business system optimization device for an autonomous and controllable environment also includes an offline strategy model training module, including:

[0053] The environment modeling submodule is used to construct the system environment model corresponding to the power business system. The system environment model includes the state space, action space, and reward function.

[0054] The network construction submodule is used to build the initial policy network and the initial value network based on the system environment model;

[0055] The interactive data determination submodule is used to obtain historical log data corresponding to the power business system, generate interactive data corresponding to the historical log data, and store the interactive data in the experience pool.

[0056] The network parameter update submodule is used to obtain batch sampling data from the experience pool and update the network parameters of the initial policy network and the initial value network based on the batch sampling data and the near-end policy optimization algorithm.

[0057] The offline policy model determination submodule is used to determine the initial policy network that meets the preset conditions as the offline policy model when the initial policy network meets the preset conditions.

[0058] Optionally, the network parameter update submodule is specifically used for:

[0059] Based on the batch sampling data, the value of the first advantage function is calculated through the first advantage function; wherein, the first advantage function is determined based on the cumulative discount reward and state value function corresponding to the batch sampling data, and the state value function is determined based on the initial value network;

[0060] Determine the probability ratio of the new and old policies corresponding to the initial policy network, and the shearing threshold corresponding to the near-end policy optimization algorithm;

[0061] Based on the first advantage function value, the probability ratio of the new and old policies, and the pruning threshold, the pruning objective function is maximized to update the network parameters of the initial policy network, thus obtaining the offline policy model.

[0062] Optionally, the network parameter update submodule is also specifically used for:

[0063] Based on state value and cumulative discounted reward, a mean squared error function is determined and minimized to update the network parameters of the initial value network, thus obtaining an offline value model.

[0064] Optionally, the power business system optimization device for an autonomous and controllable environment also includes an online strategy model training module, including:

[0065] The data acquisition submodule is used to acquire real-time feature data of the power business system based on the Prometheus monitoring system and cloud management platform, and input the real-time feature data into the offline strategy model to obtain the action probability distribution;

[0066] The value determination submodule is used to determine the value of a target action based on the action probability distribution, and to determine the value of a target state through evaluation using a target value network.

[0067] The strategy loss construction submodule is used to calculate the second advantage function based on the target state value and the target action value, and to construct the strategy loss function based on the second advantage function, the strategy entropy and the above action probability distribution.

[0068] The optimization submodule is used to construct the value loss function and optimize the offline policy model and offline value model based on the policy loss function and value loss function, respectively, to obtain the online policy model.

[0069] Optional, optimize submodules, specifically for:

[0070] The offline strategy model and / or offline value model are identified as the models to be optimized, and the current network parameters and target network parameters of the models to be optimized are determined.

[0071] The current network parameters and the target network parameters are weighted based on a preset update speed to obtain the network parameters to be updated.

[0072] Based on the network parameters to be updated, the current network parameters are gradually updated to the target network parameters.

[0073] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0074] At least one processor;

[0075] and a memory communicatively connected to the at least one processor; wherein,

[0076] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to execute the power business system optimization method for an autonomous and controllable environment as described in any embodiment of the present invention.

[0077] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the power business system optimization method for an autonomous and controllable environment as described in any embodiment of the present invention.

[0078] The technical solution of this invention involves obtaining the load volatility and resource health index of a power business system; determining a target strategy model based on the load volatility and resource health index, wherein the target strategy model includes at least one of an offline strategy model, an online strategy model, and a preset rule model; obtaining target feature data of the power business system and inputting the target feature data into the target strategy model to obtain the target resource scheduling strategy of the power business system; wherein the target feature data includes at least one of load characteristics, resource usage characteristics, configuration status characteristics, fault anomaly characteristics, and environmental disturbance characteristics. This technical solution, by switching between three models based on load volatility and resource health index, solves the technical problems of static nature, insufficient flexibility, and weak uncertainty handling capabilities in existing power business system management methods. It optimizes the scheduling strategy of power business system resources, improves the automation level of power business system scheduling management, and thus improves resource utilization, task response speed, system stability, and overall service quality.

[0079] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 A flowchart illustrating a power business system optimization method for an autonomous and controllable environment, provided as an embodiment of the present invention;

[0082] Figure 2 A framework diagram of the adaptive mechanism module provided in an embodiment of the present invention;

[0083] Figure 3 A flowchart of a preferred embodiment of the present invention is provided;

[0084] Figure 4A flowchart of another power business system optimization method for an autonomous and controllable environment provided in an embodiment of the present invention;

[0085] Figure 5 A flowchart illustrating another power business system optimization method for an autonomous and controllable environment provided by an embodiment of the present invention;

[0086] Figure 6 A schematic diagram of the structure of a power business system optimization device for an autonomous and controllable environment provided in an embodiment of the present invention;

[0087] Figure 7 A schematic diagram of the structure of an electronic device for implementing the power business system optimization method for an autonomous and controllable environment according to embodiments of the present invention. Detailed Implementation

[0088] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0089] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0090] To further clarify the technical effects of the present invention, several optimization methods for power business systems in the prior art and their existing technical problems are further explained:

[0091] 1. System optimization methods based on static rules

[0092] Traditional power system optimization methods typically rely on pre-defined rules and models. These methods adjust power system resource management based on manually designed scheduling strategies. For example, predictive models based on historical load data can provide short-term load forecasts to help dispatchers schedule power system resource allocation tasks. However, rule-based methods are usually static, suffer from high latency in human intervention, cannot adapt to sudden business fluctuations, and struggle to cover complex fault scenarios.

[0093] System optimization algorithm based on mathematical programming

[0094] Optimization algorithms for power system operations, such as linear programming, nonlinear programming, and integer programming, have been widely applied to power system business processing and resource allocation problems. These methods can optimize solutions in static environments based on constraints and optimization objectives, such as minimizing server procurement costs or optimizing container deployment locations to reduce network latency. However, these optimization algorithms often require detailed prior modeling of the system state and struggle to handle highly dynamic and uncertain situations.

[0095] Supervised learning methods based on historical data

[0096] In recent years, power system optimization techniques based on supervised learning and data-driven methods have gradually emerged. These methods primarily rely on training with large amounts of historical data to discover potential relationships between task allocation, computational demands, and workload in power systems. Common techniques include regression analysis, support vector machines, and decision trees. These methods are typically used for tasks such as workload forecasting and equipment failure prediction, but they also suffer from poor adaptability, especially in terms of limited real-time adjustment capabilities under dynamic environments.

[0097] In summary, the common drawbacks of existing technologies include: static nature, making it unable to adapt to the dynamic changes of power business systems in real time; insufficient flexibility, making it difficult to cope with complex and ever-changing environments; and weak uncertainty handling capabilities, making it difficult to effectively handle the randomness and uncertainty in power business systems.

[0098] This invention provides a power business system optimization method for autonomous and controllable environments, specifically addressing the shortcomings of existing technologies. Based on extensive historical data and utilizing reinforcement learning, this method constructs a Markov decision process for the power business system. The system can learn and progressively optimize its resource (CPU, memory, I / O, network bandwidth, etc.) scheduling strategies in a simulated environment, and adaptively adjust based on real-time feedback during actual operation. This adaptability enables the system to cope with common uncertainties and dynamic changes in power business, reducing the need for manual intervention. Furthermore, an optional online reinforcement learning scheme is provided, which can adjust strategies based on real-time feedback, ensuring the system can respond promptly to uncertain fluctuations in the power system. Compared to traditional scheduling methods, this power business system optimization method for autonomous and controllable environments improves the automation level of scheduling management, thereby enhancing resource utilization, task response speed, system stability, and overall service quality.

[0099] Figure 1 This is a flowchart illustrating a power business system optimization method for an autonomous and controllable environment, provided by an embodiment of the present invention. This embodiment is applicable to situations involving the management of system resources in a power business system. The method can be executed by a power business system optimization device for an autonomous and controllable environment. This device can be implemented in hardware and / or software and can be configured in a computer device. Figure 1 As shown, the method specifically includes the following steps:

[0100] S110, Obtain the business load volatility and resource health index of the power business system.

[0101] Among them, the business load volatility σ task This refers to indicator data that reflects the load fluctuation of the power business system. For example, the sliding standard deviation of the frequency of power business system calculation task submission can be used as the business load volatility to reflect the stability of the current business load.

[0102] Resource Health Index H sys The resource health index can be a value derived from factors such as the stability of resource utilization (e.g., CPU / GPU fluctuations) and the frequency of fault warnings. It is used to assess the overall operating status of the system. For example, the resource health index can be derived from resource usage characteristics S. resource and fault abnormality characteristics F fault The results were obtained through comprehensive calculations.

[0103] S120. Determine the target strategy model based on the business load volatility and resource health index.

[0104] The target strategy model includes at least one of an offline strategy model, an online strategy model, and a preset rule model. The offline strategy model can be a reinforcement learning model trained offline based on historical data from the power business system; the online strategy model can be a reinforcement learning model trained online based on real-time data from the power business system; and the preset rule model can be a set of rules and models based on experience, with manual execution of power business system resource scheduling and allocation based on these preset rule models.

[0105] In some embodiments, determining the target strategy model based on the business load volatility and the resource health index may include: determining the offline strategy model as the target strategy model when the business load volatility is less than a first preset volatility and the resource health index is greater than a first health index; determining the online strategy model as the target strategy model when the business load volatility is greater than the first preset volatility and less than a second preset volatility; and determining the preset rule model as the target strategy model when the business load volatility is greater than the second preset volatility or the resource health index is less than the second health index.

[0106] The first preset volatility and the second preset volatility are pre-set volatility values, with the first preset volatility being less than the second preset volatility. The first health index and the second health index are pre-set resource health values, with the first health index being greater than the second health index.

[0107] It is understandable that when the business load volatility is less than the first preset volatility and the resource health index is greater than the first health index, it means that the system load is stable and the resource status is good. The historical data has high reliability, and the offline strategy model trained offline based on the historical data can be used directly without frequent updates.

[0108] When the business load volatility is greater than a first preset volatility but less than a second preset volatility, it indicates moderate load volatility. In this case, historical data becomes partially invalid, and the strategy needs to be dynamically adjusted based on real-time data. An online strategy model trained on real-time data can be selected to adapt to changes.

[0109] When the business load volatility exceeds the second preset volatility or the resource health index is lower than the second health index, the following conditions apply. High business load volatility indicates high volatility and drastic load changes, requiring extremely high real-time performance. Reinforcement learning may not converge quickly in such cases, necessitating manually pre-defined rapid response rules. A low health index indicates frequent resource failures, requiring immediate manual intervention to prevent model misjudgments that could lead to system crashes.

[0110] For example, when σ task <5% and H sysWhen the value is greater than 0.8, an offline strategy is used, meaning the offline strategy model is used as the target strategy model. This results in higher computational efficiency and avoids the overhead of online learning. When 5% < σ task When σ is less than 15%, online fine-tuning is enabled, which means the online strategy model is determined as the target strategy model, allowing it to dynamically adapt to load changes. task ≥15% or H sys When the value is ≤0.5, switch to manually preset rules, that is, determine the preset rule model as the target policy model to ensure absolute safety and avoid the risk of reinforcement learning decision-making.

[0111] like Figure 2 The diagram shown is a framework diagram of the adaptive mechanism module provided in an embodiment of the present invention. This adaptive mechanism model can identify scenarios based on business load volatility and resource health index, determine a matching target strategy model, and switch the currently used strategy model to the target strategy model. Figure 3 The diagram shown is a flowchart of a preferred embodiment of the present invention.

[0112] S130. Obtain the target feature data of the power business system, input the target feature data into the target strategy model, and obtain the target resource scheduling strategy of the power business system.

[0113] The target feature data includes at least one of the following: service load features, resource usage features, configuration status features, fault and anomaly features, and environmental disturbance features.

[0114] Specifically, the collected target feature data is input into the target policy model, which outputs target resource scheduling policies based on the input target feature data. For example, adjusting the allocation ratio of CPU or memory, starting or stopping certain services to balance the load, triggering fault recovery mechanisms (such as switching to backup equipment), and adjusting network bandwidth allocation to cope with sudden traffic surges.

[0115] In this way, by collecting multi-dimensional target feature data from the power business system and inputting it into the target strategy model, the optimal resource scheduling strategy can be generated, thereby enabling efficient and stable operation in a dynamically changing environment.

[0116] In some embodiments, after obtaining the target resource scheduling strategy of the power business system, the following can be further performed: determine the scenario scoring weight corresponding to the power business system and the current resource scheduling strategy of the power system; weight the current resource scheduling strategy and the target resource scheduling strategy based on the scenario scoring weight to obtain a weighted resource scheduling strategy; and perform scheduling and allocation of system resources of the power business system based on the weighted resource scheduling strategy.

[0117] System resources include at least one of CPU, I / O, memory, and network bandwidth.

[0118] For example, the current strategy model is an offline strategy model, which yields the current resource scheduling strategy. The target strategy model is an online strategy model, resulting in the target resource scheduling strategy. The current and target resource scheduling strategies are then weighted based on scenario scoring weights to obtain a weighted resource scheduling strategy.

[0119] Thus, a weighted hybrid strategy is adopted to handle abrupt changes. The weight ω is dynamically adjusted according to the scenario score to achieve a smooth transition and avoid system oscillations caused by policy jumps. The value of ω ranges from 0 to 1. The final output weighted resource scheduling strategy is the weighted sum of the online and offline strategies (a final =ω·a online +(1-ω)·a offline ); where the weight coefficient ω is dynamically adjusted with the scene score (e.g., ω gradually increases from 0.2 to 0.8 in the fluctuation mode).

[0120] The technical solution of this invention involves obtaining the load volatility and resource health index of a power business system; determining a target strategy model based on the load volatility and resource health index, wherein the target strategy model includes at least one of an offline strategy model, an online strategy model, and a preset rule model; obtaining target feature data of the power business system and inputting the target feature data into the target strategy model to obtain the target resource scheduling strategy of the power business system; wherein the target feature data includes at least one of load characteristics, resource usage characteristics, configuration status characteristics, fault anomaly characteristics, and environmental disturbance characteristics. This technical solution, by switching between three models based on load volatility and resource health index, solves the technical problems of static nature, insufficient flexibility, and weak uncertainty handling capabilities in existing power business system management methods. It optimizes the scheduling strategy of power business system resources, improves the automation level of power business system scheduling management, and thus improves resource utilization, task response speed, system stability, and overall service quality.

[0121] Figure 4 This is a flowchart of another power business system optimization method for an autonomous and controllable environment provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the determination process of the offline strategy model. Figure 4 As shown, the method specifically includes the following steps:

[0122] S210. Construct system environment modeling for the power business system.

[0123] System environment modeling is a key step in reinforcement learning (RL), used to simulate the operating environment of power business systems so that agents can learn optimal policies through interaction with the environment. System environment modeling mainly includes three parts: state space, action space, and reward function.

[0124] Before system modeling, data preprocessing and feature engineering are required. Specifically, raw data is obtained from historical server system logs (database query logs, real-time task scheduling logs, service response logs, resource usage logs, etc.). To ensure data integrity and accuracy, a sliding window method is used to fill in missing values. Simultaneously, outliers are eliminated using Z-score standardization (i.e., standard score standardization), completing data cleaning. After data cleaning, the following key features are further extracted to construct the environmental state variables required for system optimization, facilitating subsequent environmental state modeling:

[0125] Business load characteristics, such as database query frequency, task submission rate, number of real-time computing tasks, and peak API calls, reflect the trend of business load changes.

[0126] Resource usage characteristics, such as CPU / GPU utilization, memory usage, and disk I / O read / write speed, are used to model resource bottlenecks.

[0127] Configuration status characteristics: Configuration metrics such as database connection pool size, cache hit rate, thread pool status, and system scheduling strategy reflect the current tuning strategy.

[0128] Anomaly and fault indicators: server alarm frequency, mean time between failures, anomaly recovery time, etc., characterize system stability risks.

[0129] Environmental disturbance characteristics: such as external request volume, changes in user access patterns, etc., characterize uncontrollable environmental disturbance factors.

[0130] The mapping between the extracted features and the state modeling of the Markov process is shown in Table 1.

[0131] Table 1

[0132]

[0133]

[0134] The specific process of power system environment modeling is as follows:

[0135] The power system environment is modeled based on Markov Decision Process (MDP), and the state space is represented as follows:

[0136] st =(P load S resource C config F fault T env )

[0137] Among them, P load S represents the current business load intensity. resource For resource usage status, C config For the current database / system configuration, F fault This indicates historical anomalies and fault fluctuation characteristics. (T) env To represent external environmental disturbances, the action space is represented as...

[0138] a t =(Δ cpu ,Δ mem ,Δ conn ,Δ cache )

[0139] Where, Δ cpu ,Δ mem This indicates adjustments to the allocation of CPU / GPU and memory resources; Δ conn ,Δ cache This indicates adjustments to configuration parameters such as database connection pool size and caching strategy; the action can be to increase, decrease, or maintain the current state. The reward function employs a multi-objective optimization design, comprehensively considering task response latency, resource utilization, and failure risk.

[0140]

[0141] Where α, β, γ are weighting parameters, and T delay T represents the response delay for the current task. max For maximum task response latency, U cpu U mem For resource utilization rate, R risk This is an indicator of abnormal risk.

[0142] S220. Based on system environment modeling, construct the initial policy network and the initial value network.

[0143] The initial policy network and the initial value network refer to the networks that have not undergone network parameter updates. The initial policy network (Actor) and the initial value network (Critic) are constructed, both of which are three-layer fully connected structures. The activation function is ReLU (Linear Correction Unit). The output layer uses the Tanh activation function to limit the action range. The discount factor is set to γ ​​= 0.99 to ensure long-term returns.

[0144] S230. Obtain historical log data corresponding to the power business system, generate interactive data corresponding to the historical log data, and store the interactive data in the experience pool.

[0145] Historical log data refers to data recording the past operating status, actions, and rewards of the power business system. Importance sampling is used in training to improve the utilization of offline data. Importance sampling is a technique for estimating expectations under different probability distributions.

[0146] A random strategy is executed in the simulation environment to generate 100,000 interaction data points (state, action, reward, and next state), which are then stored in the experience replay buffer. For example, in the simulation environment of a power business system, the state can be the current memory usage rate, fault conditions, etc., the action can be adjusting the CPU or memory allocation ratio, the reward can be calculated based on indicators such as task processing latency, and the next state is the new state of the system after the action is executed.

[0147] S240. Obtain batch sampling data from the experience pool, and update the network parameters of the initial policy network and the initial value network based on the batch sampling data and the near-end policy optimization algorithm.

[0148] Here, batch sampling data refers to batch data sampled from the experience pool. The Proximal Policy Optimization (PPO) algorithm was chosen because of its strong stability in offline policy training.

[0149] Specifically, the network parameters of the initial policy network and the initial value network can be updated based on batch sampling data and the near-end policy optimization algorithm.

[0150] In some embodiments, updating the network parameters of the initial policy network based on batch sampling data and a near-end policy optimization algorithm includes: calculating a first dominance function value based on the batch sampling data using a first dominance function; determining the probability ratio of the new and old policies corresponding to the initial policy network, and the pruning threshold corresponding to the near-end policy optimization algorithm; maximizing the pruning objective function based on the first dominance function value, the probability ratio of the new and old policies, and the pruning threshold to update the network parameters of the initial policy network and obtain an offline policy model.

[0151] The first advantage function is determined based on the cumulative discount reward and state value function corresponding to the batch sampling data, and the state value function is determined based on the initial value network.

[0152] Specifically, calculate the dominance function.

[0153] The advantage function measures the advantage of taking action at relative to the average level in the current state st; where, V(st) is the cumulative discount reward, which is the value function of state st, representing the expected cumulative reward starting from state st.

[0154] Updating the network parameters of the initial policy network can be achieved by maximizing the pruning objective function:

[0155] L CLIP (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-∈, 1+∈)A t )]

[0156] Where, r t (θ) represents the probability ratio between the old and new strategies, and ∈ = 0.2 is the shearing threshold.

[0157] In some embodiments, the method further includes: determining a mean squared error function based on state value and cumulative discounted reward, and minimizing the mean squared error function to update the network parameters of the initial value network to obtain an offline value model.

[0158] Specifically, updating the value network can be done by minimizing the mean squared error.

[0159] L VF (θ)=E t [(V θ (s t )-R t ) 2 ]

[0160] Here, Vθ(st) is the value estimate of state st by the value network with parameter θ, and Rt is the cumulative discounted reward. By minimizing this function, the value network can more accurately evaluate the value of the state. The mean squared error is minimized as the loss function to update the network parameters of the initial value network, resulting in the offline value model.

[0161] S250. When the initial policy network meets the preset conditions, the initial policy network that meets the preset conditions is determined as the offline policy model.

[0162] Specifically, the process of updating the network parameters of the initial policy network and the initial value network can be repeated. After every 10,000 iterations, the policy performance is evaluated on an independent validation set (10% of the samples drawn from historical data), the best model parameters are retained, and the best model is determined to be the model that meets the preset conditions.

[0163] For example, randomly select 10% of the historical data as a validation set, test the performance of the offline policy model on this data, and select the network model parameters with the best performance to keep.

[0164] It should also be noted that the offline policy model obtained using historical data can be used for cold start of subsequent online learning. Both are based on the Actor-Critic architecture, which facilitates refined and scenario-adaptive policy optimization based on the initial policy, breaking through the static limitations of offline policies.

[0165] S260, obtain the load volatility and resource health index of the power business system.

[0166] S270. Based on the business load volatility and resource health index, determine the target strategy model, wherein the target strategy model includes at least one of the offline strategy model, the online strategy model, and the preset rule model.

[0167] S280. Obtain the target feature data of the power business system, input the target feature data into the target strategy model, and obtain the target resource scheduling strategy of the power business system.

[0168] The target feature data includes at least one of the following: service load features, resource usage features, configuration status features, fault and anomaly features, and environmental disturbance features.

[0169] In this embodiment of the invention, by combining offline reinforcement learning and online reinforcement learning, the strategy can be dynamically adjusted to adapt to changes in the power system in real time; the optimal strategy can be automatically learned through data-driven methods to improve flexibility; and the exploration mechanism of reinforcement learning can effectively cope with the randomness and uncertainty in the power system.

[0170] Figure 5 This is a flowchart illustrating another power business system optimization method for an autonomous and controllable environment, provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the determination process of the online strategy model. For example... Figure 5 As shown, the method specifically includes the following steps:

[0171] S310, based on the Prometheus monitoring system and cloud management platform, acquires real-time feature data of the power business system and inputs the real-time feature data into the offline strategy model to obtain the action probability distribution.

[0172] Among them, the Prometheus monitoring system is used to monitor various indicators of the power business system in real time, such as resource usage (CPU, memory, disk, etc.), service response time, business load, etc.

[0173] The cloud management platform provides unified management and monitoring of cloud resources, enabling the acquisition of operational status and configuration information of power business systems within the cloud environment. These two systems allow for the acquisition of real-time characteristic data of the power business systems, reflecting their current operational status.

[0174] It is understood that real-time characteristic data can also include at least one of the following: business load characteristics, resource usage characteristics, configuration status characteristics, fault and anomaly characteristics, and environmental disturbance characteristics. Real-time characteristic data is determined in real time.

[0175] The acquired real-time feature data can be input into a previously trained offline policy model. This offline policy model, trained on historical data, can output a probability distribution of actions based on the input state information. That is, it calculates and outputs the probability of each possible action to form an action probability distribution. Here, the action probability distribution represents the likelihood of taking each action in the current state.

[0176] It should also be noted that, in order to improve data quality, the Isolation Forest algorithm is used to filter outliers and noisy data, and the state vector (i.e., each real-time feature data) is updated every 5 seconds to drive the online adjustment of the strategy.

[0177] S320. Based on the action probability distribution, determine the target action value, and evaluate and determine the target state value through the target value network.

[0178] The action probability distribution is denoted as π(a|s); the target value network can be an offline value model obtained through previous offline training. The target state value is denoted as V(s).

[0179] S330. Calculate the second advantage function based on the target state value and the target action value, and construct the policy loss function based on the second advantage function, policy entropy, and the above action probability distribution.

[0180] Specifically, the offline policy model is further updated based on the Actor-Critic architecture. The Actor network receives the current state s. t The output action probability distribution π(a|s) is calculated; the Criti c network evaluates the state value V(s) and calculates the second advantage function A'(s,a)=Q(s,a)-V(s), where V(s) is the state value; Q(s,a) is the action value, which evaluates the long-term expected benefit of performing action a in state s.

[0181] The policy loss function is the weighted sum of the advantage function and the policy entropy:

[0182] L Actor =-E[logπ(a) t |s t )·A'(s t ,a t )+αH(π(·|s t))]

[0183] Where α = 0.2 is the entropy weighting coefficient, which encourages strategy exploration. L Actor The loss function of the Actor network is used to optimize the policy by maximizing this function. logπ(a t |s t ): In state s t Take action a t The logarithmic probability of A'(s). t ,a t ): Take action a in state st. t The advantage function represents the superiority or inferiority of an action relative to the average policy. H(π(·∣st)): The advantage function of policy π in state s. t Entropy is used to measure the randomness of a strategy.

[0184] S340. Construct a value loss function, and optimize the offline policy model and offline value model based on the policy loss function and value loss function respectively to obtain the online policy model.

[0185] The value loss function minimizes the value prediction error.

[0186] L Critic =E[(Q(s) t ,a t )-(r t +γV(s t+1 )) 2 ]

[0187] Among them, L Critic : The loss function of the Critic network, which is used to optimize value estimation.

[0188] Q(s t ,a t ): In state s t Take action a t The actual value of the action.

[0189] r t In state s t Take action a t The instant reward received afterward.

[0190] γ: Discount factor, used to balance the importance of immediate rewards and future rewards.

[0191] V(s t+1 ): In state s t+1 The value of the state under the given conditions.

[0192] The loss function measures the action value Q(s) predicted by the Critic network. t ,at The target value r is calculated based on the immediate reward and the value of the next state. t +γV(s t+1 By minimizing the error between states and actions, the Critic network can more accurately estimate the value of states and actions.

[0193] In another implementation, the policy can be updated every 30 minutes, using a soft update method to update the parameters of the policy network (Actor) and the value network:

[0194] θ' Actor ←τθ Actor +(1-τ)θ' Actor

[0195] θ' Critic ←τθ Critic +(1-τ)θ' Critic

[0196] The update rate is set to τ = 0.001. Actor θ': Parameters of the original Actor network. Actor : The parameters of the target Actor network (updated parameters). The formula indicates that the parameter update of the target Actor network is a weighted average of the original Actor network parameters and the original target Actor network parameters. By using a small τ value, the target network parameters are updated slowly, which helps to stabilize the training process.

[0197] θ Critic θ': Parameters of the original Critic network. Critic : The parameters of the target Critic network (updated parameters). Similarly, this formula updates the target Critic network parameters using a weighted average to achieve stable training.

[0198] It should also be noted that this invention can set a security constraint mechanism. Specifically, an action masking mechanism is used to prohibit actions that exceed the device's security threshold, such as prohibiting increasing the service load when the service load rate is >95%; sampling reward plasticity applies a penalty term r to actions that violate the constraints. penalty = -10 times the number of violations.

[0199] S350, obtain the load volatility and resource health index of the power business system.

[0200] S360. Determine the target strategy model based on business load volatility and resource health index.

[0201] The target strategy model includes at least one of the following: offline strategy model, online strategy model, and preset rule model.

[0202] S370. Obtain the target feature data of the power business system, input the target feature data into the target strategy model, and obtain the target resource scheduling strategy of the power business system.

[0203] The target feature data includes at least one of the following: service load features, resource usage features, configuration status features, fault and anomaly features, and environmental disturbance features.

[0204] In this embodiment of the invention, a real-time data-driven dynamic optimization strategy is employed to address the inherent limitations of offline reinforcement learning in terms of dynamic environment adaptability and real-time security: the real-time state of power business systems cannot be fully covered by historical data, and offline strategies may fail due to environmental changes; the simplified assumptions of simulation models regarding the real environment can lead to deviations in actual strategy deployment. The core differences between offline and online reinforcement learning are shown in Table 2.

[0205] Table 2

[0206]

[0207] This invention combines phased offline pre-training with online fine-tuning. Offline learning provides a globally optimal basic strategy, while online learning achieves local real-time optimization. The two are dynamically integrated through an adaptive mechanism, enabling dynamic adaptability in power system optimization while ensuring strategy security. The offline module constructs a multi-objective optimization strategy based on the PPO algorithm, the online module achieves real-time fine-tuning through an Actor-Critic architecture and security constraint mechanisms, and the adaptive module intelligently switches strategy modes according to system state. The three work together to overcome the technical bottlenecks of traditional methods in terms of staticity, flexibility, and uncertainty handling capabilities.

[0208] This invention introduces a fusion mechanism of offline and online reinforcement learning. The system can pre-train strategies using historical data and dynamically adjust them through real-time feedback, solving the problem that traditional static rules are unable to cope with load fluctuations and equipment failures. This significantly shortens the response time for business queries and calculations, effectively improving the stability of the power business system. Through the design of a multi-objective reward function, the system simultaneously optimizes business scheduling efficiency, cost, and equipment wear during the Markov decision process, extending the lifespan of critical resource equipment and overcoming the limitations of traditional single-objective optimization algorithms. Based on action masking and reward shaping mechanisms, the system forces policies to meet the physical constraints of the power business system, reducing violations and solving the key challenge of reinforcement learning potentially causing security risks in complex systems. Ultimately, this invention achieves optimized scheduling strategies for power business system resources, improves the automation level of power business system scheduling management, and thus improves resource utilization, task response speed, system stability, and overall service quality.

[0209] Figure 6 This is a schematic diagram of a power business system optimization device for an autonomous and controllable environment, provided as an embodiment of the present invention. Figure 6 As shown, the device includes:

[0210] The load and health index determination module 410 is used to obtain the business load volatility and resource health index of the power business system.

[0211] The target strategy model determination module 420 is used to determine the target strategy model based on the business load volatility and resource health index, wherein the target strategy model includes at least one of the offline strategy model, online strategy model and preset rule model;

[0212] The resource scheduling strategy generation module 430 is used to acquire target feature data of the power business system, input the target feature data into the target strategy model, and obtain the target resource scheduling strategy of the power business system.

[0213] The target feature data includes at least one of the following: service load features, resource usage features, configuration status features, fault and anomaly features, and environmental disturbance features.

[0214] Optionally, the target strategy model determination module 420 is specifically used for:

[0215] If the business load volatility is less than the first preset volatility and the resource health index is greater than the first health index, the offline strategy model will be determined as the target strategy model.

[0216] If the volatility of the business load is greater than the first preset volatility but less than the second preset volatility, the online strategy model will be determined as the target strategy model.

[0217] If the business load volatility is greater than the second preset volatility or the resource health index is less than the second health index, the preset rule model will be determined as the target strategy model.

[0218] Optionally, the power business system optimization device for an autonomous and controllable environment also includes a dispatch module, specifically used for:

[0219] After obtaining the target resource scheduling strategy of the power business system, determine the scenario scoring weight corresponding to the power business system, as well as the current resource scheduling strategy of the power system;

[0220] The current resource scheduling strategy and the target resource scheduling strategy are weighted based on the scenario scoring weights to obtain the weighted resource scheduling strategy.

[0221] Based on a weighted resource scheduling strategy, the system resources of the power business system are scheduled and allocated. The system resources include at least one of CPU, I / O, memory and network bandwidth.

[0222] Optionally, the power business system optimization device for an autonomous and controllable environment also includes an offline strategy model training module, including:

[0223] The environment modeling submodule is used to construct the system environment model corresponding to the power business system. The system environment model includes the state space, action space, and reward function.

[0224] The network construction submodule is used to build the initial policy network and the initial value network based on the system environment model;

[0225] The interactive data determination submodule is used to obtain historical log data corresponding to the power business system, generate interactive data corresponding to the historical log data, and store the interactive data in the experience pool.

[0226] The network parameter update submodule is used to obtain batch sampling data from the experience pool and update the network parameters of the initial policy network and the initial value network based on the batch sampling data and the near-end policy optimization algorithm.

[0227] The offline policy model determination submodule is used to determine the initial policy network that meets the preset conditions as the offline policy model when the initial policy network meets the preset conditions.

[0228] Optionally, the network parameter update submodule is specifically used for:

[0229] Based on the batch sampling data, the value of the first advantage function is calculated through the first advantage function; wherein, the first advantage function is determined based on the cumulative discount reward and state value function corresponding to the batch sampling data, and the state value function is determined based on the initial value network;

[0230] Determine the probability ratio of the new and old policies corresponding to the initial policy network, and the shearing threshold corresponding to the near-end policy optimization algorithm;

[0231] Based on the first advantage function value, the probability ratio of the new and old policies, and the pruning threshold, the pruning objective function is maximized to update the network parameters of the initial policy network, thus obtaining the offline policy model.

[0232] Optionally, the network parameter update submodule is also specifically used for:

[0233] Based on state value and cumulative discounted reward, a mean squared error function is determined and minimized to update the network parameters of the initial value network, thus obtaining an offline value model.

[0234] Optionally, the power business system optimization device for an autonomous and controllable environment also includes an online strategy model training module, including:

[0235] The data acquisition submodule is used to acquire real-time feature data of the power business system based on the Prometheus monitoring system and cloud management platform, and input the real-time feature data into the offline strategy model to obtain the action probability distribution;

[0236] The value determination submodule is used to determine the value of a target action based on the action probability distribution, and to determine the value of a target state through evaluation using a target value network.

[0237] The strategy loss construction submodule is used to calculate the second advantage function based on the target state value and the target action value, and to construct the strategy loss function based on the second advantage function, the strategy entropy and the above action probability distribution.

[0238] The optimization submodule is used to construct the value loss function and optimize the offline policy model and offline value model based on the policy loss function and value loss function respectively to obtain the online policy model.

[0239] Optional, optimize submodules, specifically for:

[0240] The offline strategy model and / or offline value model are identified as the models to be optimized, and the current network parameters and target network parameters of the models to be optimized are determined.

[0241] The current network parameters and the target network parameters are weighted based on a preset update speed to obtain the network parameters to be updated.

[0242] Based on the network parameters to be updated, the current network parameters are gradually updated to the target network parameters.

[0243] The power business system optimization device for an autonomous and controllable environment provided in this embodiment of the invention can execute the power business system optimization method for an autonomous and controllable environment provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0244] Figure 7 This is a schematic diagram of an electronic device used to implement the power business system optimization method for an autonomous and controllable environment according to embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0245] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0246] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0247] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the optimization of power business systems for autonomous and controllable environments.

[0248] In some embodiments, the optimization of a power business system for an autonomous and controllable environment can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the power business system optimization for an autonomous and controllable environment described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform power business system optimization for an autonomous and controllable environment by any other suitable means (e.g., by means of firmware).

[0249] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0250] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0251] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0252] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0253] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0254] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0255] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0256] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for optimizing a power business system in an autonomous and controllable environment, characterized in that, include: Obtain the load volatility and resource health index of the power business system; where load volatility refers to the indicator data reflecting the load volatility of the power business system. Based on the business load volatility and resource health index, a target strategy model is determined, wherein the target strategy model includes at least one of an offline strategy model, an online strategy model, and a preset rule model; Obtain the target feature data of the power business system, input the target feature data into the target strategy model, and obtain the target resource scheduling strategy of the power business system; The target feature data includes at least one of the following: service load features, resource usage features, configuration status features, fault and anomaly features, and environmental disturbance features. The step of determining the target strategy model based on the business load volatility and resource health index includes: If the business load volatility is less than the first preset volatility and the resource health index is greater than the first health index, the offline strategy model will be determined as the target strategy model. If the business load volatility is greater than the first preset volatility and less than the second preset volatility, the online strategy model is determined as the target strategy model. If the business load volatility is greater than the second preset volatility or the resource health index is less than the second health index, the preset rule model will be determined as the target strategy model. The resource health index is a value obtained by integrating the stability of resource utilization and the frequency of fault warnings. The resource health index is used to evaluate the overall operating status of the system and is calculated by combining resource usage characteristics and fault anomaly characteristics.

2. The method according to claim 1, characterized in that, After obtaining the target resource scheduling strategy of the power business system, the method further includes: Determine the scenario scoring weights corresponding to the power business system, and the current resource scheduling strategy of the power system; The current resource scheduling strategy and the target resource scheduling strategy are weighted based on the scenario scoring weights to obtain a weighted resource scheduling strategy. Based on the weighted resource scheduling strategy, the system resources of the power business system are scheduled and allocated, wherein the system resources include at least one of CPU, I / O, memory and network bandwidth.

3. The method according to claim 1, characterized in that, The training process of the offline policy model includes: Construct a system environment model corresponding to the power business system, wherein the system environment model includes a state space, an action space, and a reward function; Based on the system environment modeling, construct the initial policy network and the initial value network; Obtain historical log data corresponding to the power business system, generate interactive data corresponding to the historical log data, and store the interactive data in the experience pool; Batch sampling data is obtained from the experience pool, and the network parameters of the initial policy network and the initial value network are updated based on the batch sampling data and the near-end policy optimization algorithm. When the initial policy network meets the preset conditions, the initial policy network that meets the preset conditions is determined as the offline policy model.

4. The method according to claim 3, characterized in that, The step of updating the network parameters of the initial policy network based on the batch sampling data and the near-end policy optimization algorithm includes: Based on the batch sampling data, a first advantage function value is calculated using a first advantage function; wherein, the first advantage function is determined based on the cumulative discount reward and state value function corresponding to the batch sampling data, and the state value function is determined based on the initial value network; Determine the probability ratio of the new and old policies corresponding to the initial policy network, and the shearing threshold corresponding to the near-end policy optimization algorithm; Based on the first advantage function value, the probability ratio of the new and old policies, and the pruning threshold, the pruning objective function is maximized to update the network parameters of the initial policy network, thereby obtaining the offline policy model.

5. The method according to claim 4, characterized in that, The method further includes: Based on the state value and the cumulative discount reward, a mean squared error function is determined and minimized to update the network parameters of the initial value network, thereby obtaining an offline value model.

6. The method according to claim 1, characterized in that, The training process of the online policy model includes: Based on the Prometheus monitoring system and cloud management platform, real-time feature data of the power business system is obtained, and the real-time feature data is input into the offline strategy model to obtain the action probability distribution. Based on the action probability distribution, the target action value is determined, and the target state value is determined through evaluation using a target value network; A second advantage function is calculated based on the target state value and the target action value, and a policy loss function is constructed based on the second advantage function, the policy entropy, and the action probability distribution. A value loss function is constructed, and the offline policy model and the offline value model are optimized based on the policy loss function and the value loss function, respectively, to obtain the online policy model.

7. The method according to claim 6, characterized in that, The optimization of the offline policy model and the offline value model based on the policy loss function and the value loss function, respectively, includes: The offline strategy model and / or the offline value model are identified as the models to be optimized, and the current network parameters and target network parameters of the models to be optimized are determined. The current network parameters and the target network parameters are weighted based on a preset update rate to obtain the network parameters to be updated. Based on the network parameters to be updated, the current network parameters are gradually updated to the target network parameters.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the power business system optimization method for an autonomous and controllable environment as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the power business system optimization method for an autonomous and controllable environment as described in any one of claims 1-7.

Images