Network scheduling model training method, device, equipment, medium and product

Abstract

The invention relates to the technical field of machine learning, and particularly provides a network scheduling model training method and device, equipment, a medium and a product. The method comprises the following steps: generating offline training data according to a Lagrange multiplier, a plurality of network scheduling state samples and network scheduling actions respectively corresponding to the network scheduling state samples; performing parameter adjustment on the reinforcement learning model according to the offline training data; updating a Lagrange multiplier according to the adjusted reinforcement learning model; and if it is determined that the updated Lagrangian multiplier meets a convergence condition, taking the adjusted reinforcement learning model as a trained target network scheduling model, otherwise, updating off-line training data according to the updated Lagrangian multiplier, and adjusting the reinforcement learning model based on the updated off-line training data. Therefore, model training is carried out through the continuously updated offline training data, and the training cost can be reduced while the model performance is considered.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and in particular to a method, device, equipment, medium and product for training a network scheduling model. Background Art

[0002] In network scheduling scenarios (such as instant messaging scenarios), in order to meet the multi-user delay constraint requirements and resource constraint requirements, it is usually necessary to interact with the online network system in real time during the training phase and perform model training based on the real-time data of the network system so that the network system can be scheduled through the trained model.

[0003] However, using this real-time interactive approach to model training usually consumes a lot of service costs. If offline training data is used for model training, the performance of the trained model is usually poor. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a method, device, equipment, medium and product for training a network scheduling model, so as to reduce the training cost while taking into account the model performance when training the network scheduling model.

[0005] On the one hand, an embodiment of the present disclosure provides a method for training a network scheduling model, including:

[0006] Offline training data is generated according to the Lagrange multiplier, a plurality of network scheduling state samples and their corresponding network scheduling actions; the offline training data includes a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards; the reward is determined according to the corresponding network scheduling state samples and network scheduling actions, and the Lagrange multiplier;

[0007] Adjust the parameters of the reinforcement learning model based on offline training data;

[0008] Update the Lagrange multiplier according to the adjusted reinforcement learning model;

[0009] If it is determined that the updated Lagrange multiplier meets the convergence condition, the adjusted reinforcement learning model is used as the trained target network scheduling model. Otherwise, the offline training data is updated according to the updated Lagrange multiplier, and the reinforcement learning model is adjusted based on the updated offline training data.

[0010] In one implementation, adjusting parameters of the reinforcement learning model based on offline training data includes:

[0011] Determine the strategy target value according to multiple network scheduling state samples and their corresponding network scheduling actions and rewards;

[0012] According to the strategy target value, the model parameters of the reinforcement learning model are preliminarily adjusted;

[0013] According to the iteration step, the target strategy network is updated to obtain an updated target strategy network;

[0014] The initially adjusted reinforcement learning model is adjusted again according to the updated targeted policy network.

[0015] In one implementation, determining a strategy target value based on a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards includes:

[0016] Adopting an action value function, according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards, determining the action value function value corresponding to each network scheduling state sample;

[0017] The strategy objective function is adopted to determine the strategy objective value according to multiple network scheduling state samples and their corresponding network scheduling actions, rewards and action value function values.

[0018] In one embodiment, before using the action-value function to determine the action-value function value corresponding to each network scheduling state sample according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards, the method further includes:

[0019] Adopting the action objective function, determining the action objective function value according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards;

[0020] According to the value of the action objective function, the action value function is updated.

[0021] In one implementation, updating the Lagrange multiplier according to the adjusted reinforcement learning model includes:

[0022] Predict network scheduling actions based on the collected network scheduling status and the adjusted reinforcement learning model;

[0023] Determine the network scheduling energy consumption according to the network scheduling state and its corresponding network scheduling action;

[0024] The Lagrangian duality and energy constraints are adopted to consume energy according to network scheduling and update the Lagrangian multiplier.

[0025] In one implementation, determining that the updated Lagrange multiplier meets the convergence condition includes:

[0026] determining a difference between the Lagrange multiplier before the update and the Lagrange multiplier after the update;

[0027] If the difference is lower than the convergence threshold, it is determined that the convergence condition is met.

[0028] On the one hand, an embodiment of the present disclosure provides a device for training a network scheduling model, including:

[0029] A generating unit is used to generate offline training data according to the Lagrange multiplier, a plurality of network scheduling state samples and their corresponding network scheduling actions; the offline training data includes a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards; the reward is determined according to the corresponding network scheduling state samples and network scheduling actions, and the Lagrange multiplier;

[0030] An adjustment unit, used to adjust parameters of the reinforcement learning model according to offline training data;

[0031] An updating unit, used to update the Lagrange multiplier according to the adjusted reinforcement learning model;

[0032] The determination unit is used to use the adjusted reinforcement learning model as the trained target network scheduling model if it is determined that the updated Lagrange multiplier meets the convergence condition; otherwise, update the offline training data according to the updated Lagrange multiplier, and adjust the reinforcement learning model based on the updated offline training data.

[0033] In one implementation, the adjustment unit is used to:

[0034] Determine the strategy target value according to multiple network scheduling state samples and their corresponding network scheduling actions and rewards;

[0035] According to the strategy target value, the model parameters of the reinforcement learning model are preliminarily adjusted;

[0036] According to the iteration step, the target strategy network is updated to obtain an updated target strategy network;

[0037] The initially adjusted reinforcement learning model is adjusted again according to the updated targeted policy network.

[0038] In one implementation, the adjustment unit is used to:

[0039] Adopting an action value function, according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards, determining the action value function value corresponding to each network scheduling state sample;

[0040] The strategy objective function is adopted to determine the strategy objective value according to multiple network scheduling state samples and their corresponding network scheduling actions, rewards and action value function values.

[0041] In one implementation, the adjustment unit is further configured to:

[0042] Adopting the action objective function, determining the action objective function value according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards;

[0043] According to the value of the action objective function, the action value function is updated.

[0044] In one implementation, the updating unit is used to:

[0045] Predict network scheduling actions based on the collected network scheduling status and the adjusted reinforcement learning model;

[0046] Determine the network scheduling energy consumption according to the network scheduling state and its corresponding network scheduling action;

[0047] The Lagrangian duality and energy constraints are adopted to consume energy according to network scheduling and update the Lagrangian multiplier.

[0048] In one implementation, the determining unit is used to:

[0049] determining a difference between the Lagrange multiplier before the update and the Lagrange multiplier after the update;

[0050] If the difference is lower than the convergence threshold, it is determined that the convergence condition is met.

[0051] In one aspect, an electronic device is provided in an embodiment of the present disclosure, including:

[0052] Processor; and

[0053] A memory stores computer instructions, wherein the computer instructions are used to enable a processor to execute the steps of the method provided in any of the various optional implementations of any of the above-mentioned network scheduling model training.

[0054] On the one hand, a computer-readable storage medium is provided in an embodiment of the present disclosure, storing computer instructions for causing a computer to execute the steps of a method provided in any of the various optional implementations of the network scheduling model training described above.

[0055] On the one hand, a computer program product is provided in an embodiment of the present disclosure, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the steps of the method provided in any of the various optional implementations of the above-mentioned network scheduling model training.

[0056] The method for training a network scheduling model in the disclosed embodiment includes generating offline training data according to Lagrange multipliers, multiple network scheduling state samples and their corresponding network scheduling actions; the offline training data includes multiple network scheduling state samples and their corresponding network scheduling actions and rewards; the rewards are determined according to the corresponding network scheduling state samples and network scheduling actions, as well as Lagrange multipliers; adjusting the parameters of the reinforcement learning model according to the offline training data; updating the Lagrange multipliers according to the adjusted reinforcement learning model; if it is determined that the updated Lagrange multipliers meet the convergence conditions, the adjusted reinforcement learning model is used as the trained target network scheduling model, otherwise, the offline training data is updated according to the updated Lagrange multipliers, and the reinforcement learning model is adjusted based on the updated offline training data. In this way, when training the network scheduling model, model training is performed based on offline training data, and real-time interaction with the network system is not required. By continuously updating the offline training data and model parameters, the quality of the training data is guaranteed, so that the training cost can be reduced while taking into account the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of a method for training a network scheduling model in an embodiment of the present disclosure.

[0058] Figure 2 It is a schematic diagram of a model training process in an embodiment of the present disclosure.

[0059] Figure 3 It is a flow chart of a network scheduling method in an embodiment of the present disclosure.

[0060] Figure 4 It is a structural block diagram of a device for training a network scheduling model in an embodiment of the present disclosure.

[0061] Figure 5 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] The technical solution of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described implementation is a part of the implementation of the present disclosure, rather than all the implementations. Based on the implementation in the present disclosure, all other implementations obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure. In addition, the technical features involved in the different implementations of the present disclosure described below can be combined with each other as long as they do not conflict with each other.

[0063] In the scenario of network scheduling model training, if real-time interaction is used for model training, it usually consumes a lot of service costs. If offline training data is used for model training, the performance of the trained model is usually poor due to problems such as uneven distribution of training data and limited generalization ability.

[0064] Based on the defects of the above-mentioned related technologies, the embodiments of the present disclosure provide a method, device, equipment, medium and product for training a network scheduling model, aiming to reduce training costs while taking into account model performance when training a network scheduling model.

[0065] A method for training a network scheduling model is provided in an embodiment of the present disclosure. The method can be applied to electronic devices. The present disclosure does not limit the type of electronic device. It can be any type of device suitable for implementation, such as a terminal device and a server, etc. The present disclosure will not go into details about this.

[0066] See also Figure 1 As shown, it is a flowchart of a method for training a network scheduling model in an embodiment of the present disclosure. Figure 1 The method is described below. The specific implementation process of the method is as follows:

[0067] Step 101: Generate offline training data according to Lagrange multipliers, a plurality of network scheduling state samples and their corresponding network scheduling actions.

[0068] In one implementation, an initial data set and a Lagrange multiplier are obtained, wherein the initial data set includes a plurality of network scheduling state samples and their corresponding network scheduling actions, and a reward function is used to determine the rewards corresponding to each network scheduling state sample according to the Lagrange multiplier and the initial data set.

[0069] The offline training data includes multiple network scheduling state samples and their corresponding network scheduling actions and rewards; the reward is determined based on the corresponding network scheduling state samples and network scheduling actions, as well as the Lagrange multiplier. The reward is the feedback from the environment after the network scheduling action is executed. Optionally, the reward function can be set according to the actual application scenario to determine the reward.

[0070] In the embodiments of the present disclosure, it can be applied to a network scheduling scenario. The network scheduling status sample can be a system indicator of a network system that needs to be scheduled, and the system indicator can be set according to the actual application scenario.

[0071] Step 102: Adjust parameters of the reinforcement learning model according to the offline training data.

[0072] Among them, the reinforcement learning model is a strategy network based on reinforcement learning, which can be expressed as π θ(a|s), where s represents the network scheduling state (e.g., it can be a network scheduling state sample), and a represents the network scheduling action.

[0073] In one implementation, when executing step 102, the following steps may be used:

[0074] S1021: Determine a strategy target value based on a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards.

[0075] In one implementation, an action value function is used to determine the action value function value corresponding to each network scheduling state sample based on multiple network scheduling state samples and their corresponding network scheduling actions and rewards; a strategy objective function is used to determine the strategy objective value based on multiple network scheduling state samples and their corresponding network scheduling actions, rewards and action value function values.

[0076] Among them, the action value function can be expressed as Q φ (s, a). The training of the action value function is performed based on the Constrained Q-Learning (CQL) algorithm based on the Soft Actor-Critic (SAC) algorithm, and the training of the reinforcement learning model obeys the action correction algorithm. The action value function is used to determine the action value function value, and the policy objective function is used to determine the policy objective value. Both the action value function and the policy objective function can be set according to the actual application scenario, and there is no restriction here.

[0077] Furthermore, the action-value function can also be updated.

[0078] In one implementation, an action objective function is used to determine an action objective function value based on a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards; and an action value function is updated based on the action objective function value.

[0079] In this way, the action-value function can be updated during model training.

[0080] S1022: Preliminary adjustment of model parameters of the reinforcement learning model according to the strategy target value.

[0081] S1023: updating the targeting strategy network according to the iteration step corresponding to the targeting strategy network to obtain an updated targeting strategy network.

[0082] In practical applications, the iteration step size corresponding to the targeted policy network can be set according to the actual application scenario, and the update method of the targeted policy network can be set according to the actual application scenario, and there is no restriction here. The targeted policy network is used to fine-tune the model parameters of the reinforcement learning model.

[0083] S1024: According to the updated target strategy network, the reinforcement learning model after the initial adjustment is adjusted again.

[0084] Since the offline training data is usually large in quantity, the offline training data can be divided into T parts, each of which is a batch of data. T is a positive integer, such as 10. Then, for each batch of data, the following steps are performed: Execute S1021-S1024 based on the batch data.

[0085] In this way, the optimal reinforcement learning model under the offline training data can be obtained.

[0086] Step 103: Update the Lagrange multiplier according to the adjusted reinforcement learning model.

[0087] In one implementation, the network scheduling action is predicted based on the collected network scheduling state and the adjusted reinforcement learning model; the network scheduling energy consumption is determined based on the network scheduling state and its corresponding network scheduling action; and the Lagrangian duality and energy constraints are used to update the Lagrangian multiplier based on the network scheduling energy consumption.

[0088] As an example, a batch of trajectory data is collected, and the trajectory data contains multiple network scheduling states. The network scheduling state is input into the adjusted reinforcement learning model to obtain the predicted optimal network scheduling action. Then, the collected network scheduling state and its corresponding network scheduling action can be input into the energy function to obtain the network scheduling consumption energy, which represents the energy consumed by executing the network scheduling action. Further, the iteration step size corresponding to the Lagrange multiplier is obtained, and the Lagrange duality and energy constraints are used to update the Lagrange multiplier according to the network scheduling consumption energy and the iteration step size.

[0089] Optionally, both the energy function and the energy constraint condition may be set according to the actual application scenario. For example, the energy constraint condition may be lower than an energy threshold, which is not limited here.

[0090] Step 104: If it is determined that the updated Lagrange multiplier meets the convergence condition, the adjusted reinforcement learning model is used as the trained target network scheduling model; otherwise, the offline training data is updated according to the updated Lagrange multiplier, and the reinforcement learning model is adjusted based on the updated offline training data.

[0091] In one implementation, the difference between the Lagrangian multiplier before the update and the Lagrangian multiplier after the update is determined; if the difference is lower than a convergence threshold, it is determined that the convergence condition is met; otherwise, it is determined that the convergence condition is not met.

[0092] In this way, if it is determined that the convergence conditions are met, a trained target network scheduling model can be obtained.

[0093] Combine the following Figure 2 This section provides an example of the training process of the target network scheduling model. Figure 2 ,shown is a schematic diagram of a model training process.

[0094] The initial data set includes multiple network scheduling state samples s and their corresponding network scheduling actions a. Based on the initial data set, the following steps can be executed in a loop until a trained target network scheduling model is obtained: according to the Lagrange multiplier and the initial data set, the reward r corresponding to each network scheduling state sample is generated (or updated), the offline training data is obtained, and the model parameters of the reinforcement learning model are adjusted based on the offline training data to obtain the adjusted reinforcement learning model, and the Lagrange multiplier is updated based on the adjusted reinforcement learning model.

[0095] Furthermore, network scheduling based on the target network scheduling model can be performed in an online network system environment.

[0096] Combine the following Figure 3 This paper describes the network scheduling based on the target network scheduling model. Figure 3 FIG. 1 is a flowchart of a network scheduling method in an embodiment of the present disclosure. The specific implementation process of the method is as follows:

[0097] Step 301: Obtain the network scheduling status of the network system to be scheduled.

[0098] Step 302: Input the network scheduling state into the target network scheduling model to obtain the network scheduling action.

[0099] Step 303: Execute network scheduling action.

[0100] In traditional methods, real-time interaction with the actual network system is usually required during model training, which obviously reduces system performance and generates huge service costs, making model training more difficult. In the disclosed embodiment, a new offline algorithm tailored for multi-user delay constraint scheduling is adopted, namely, the Scheduled Offline Learning And Rectification (SOLAR) scheduling algorithm. During the training process, an offline learning mechanism is used to perform effective learning scheduling only from pre-collected offline training data. No real-time system interaction is required, which reduces the impact on system performance and reduces the cost of model training.

[0101] Furthermore, in the traditional method, there are usually problems such as training data distribution bias, limited generalization ability and low sample efficiency. In the embodiment of the present disclosure, by adjusting the Lagrange multiplier, updating the reward in the offline training data, and then updating the reinforcement learning model, the trained target network scheduling model can obtain ideal performance under energy constraints, which exceeds the level of those that only rely on the initial data set strategy, especially in those fields that are extremely sensitive to environmental interactions and have extremely high requirements. In addition, by updating the Lagrange multiplier through the Lagrange duality, the delay-sensitive queue environment problem is solved, and compliance with resource constraints and delay constraints is ensured in a completely offline manner. It can be applied to a variety of network systems and has a wide range of applications.

[0102] Based on the same inventive concept, a device for training a network scheduling model is also provided in an embodiment of the present disclosure. Since the principle of solving the problem by the above-mentioned device and equipment is similar to that of a method for training a network scheduling model, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present disclosure does not limit the type of electronic device. It can be any type of device suitable for implementation, such as terminal devices and servers, etc. The present disclosure will not repeat this. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution.

[0103] See also Figure 4 , which is a structural block diagram of a network scheduling model training device in an embodiment of the present disclosure. In some embodiments, the network scheduling model training device of the present disclosure example includes:

[0104] A generating unit 401 is used to generate offline training data according to the Lagrange multiplier, a plurality of network scheduling state samples and their corresponding network scheduling actions; the offline training data includes a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards; the reward is determined according to the corresponding network scheduling state samples and network scheduling actions, and the Lagrange multiplier;

[0105] An adjustment unit 402, used to adjust parameters of the reinforcement learning model according to offline training data;

[0106] An updating unit 403, used for updating the Lagrange multiplier according to the adjusted reinforcement learning model;

[0107] The determination unit 404 is used to use the adjusted reinforcement learning model as the trained target network scheduling model if it is determined that the updated Lagrange multiplier meets the convergence condition; otherwise, update the offline training data according to the updated Lagrange multiplier and adjust the reinforcement learning model based on the updated offline training data.

[0108] In one implementation, the adjustment unit 402 is used to:

[0109] Determine the strategy target value according to multiple network scheduling state samples and their corresponding network scheduling actions and rewards;

[0110] According to the strategy target value, the model parameters of the reinforcement learning model are preliminarily adjusted;

[0111] According to the iteration step, the target strategy network is updated to obtain an updated target strategy network;

[0112] The initially adjusted reinforcement learning model is adjusted again according to the updated targeted policy network.

[0113] In one implementation, the adjustment unit 402 is used to:

[0114] Adopting an action value function, according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards, determining the action value function value corresponding to each network scheduling state sample;

[0115] The strategy objective function is adopted to determine the strategy objective value according to multiple network scheduling state samples and their corresponding network scheduling actions, rewards and action value function values.

[0116] In one implementation, the adjustment unit 402 is further configured to:

[0117] Adopting the action objective function, determining the action objective function value according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards;

[0118] According to the value of the action objective function, the action value function is updated.

[0119] In one implementation, the updating unit 403 is used to:

[0120] Predict network scheduling actions based on the collected network scheduling status and the adjusted reinforcement learning model;

[0121] Determine the network scheduling energy consumption according to the network scheduling state and its corresponding network scheduling action;

[0122] The Lagrangian duality and energy constraints are adopted to consume energy according to network scheduling and update the Lagrangian multiplier.

[0123] In one implementation, the determining unit 404 is configured to:

[0124] determining a difference between the Lagrange multiplier before the update and the Lagrange multiplier after the update;

[0125] If the difference is lower than the convergence threshold, it is determined that the convergence condition is met.

[0126] The method for training a network scheduling model in the disclosed embodiment includes generating offline training data according to Lagrange multipliers, multiple network scheduling state samples and their corresponding network scheduling actions; the offline training data includes multiple network scheduling state samples and their corresponding network scheduling actions and rewards; the rewards are determined according to the corresponding network scheduling state samples and network scheduling actions, as well as Lagrange multipliers; adjusting the parameters of the reinforcement learning model according to the offline training data; updating the Lagrange multipliers according to the adjusted reinforcement learning model; if it is determined that the updated Lagrange multipliers meet the convergence conditions, the adjusted reinforcement learning model is used as the trained target network scheduling model, otherwise, the offline training data is updated according to the updated Lagrange multipliers, and the reinforcement learning model is adjusted based on the updated offline training data. In this way, when training the network scheduling model, model training is performed based on offline training data, and real-time interaction with the network system is not required. By continuously updating the offline training data and model parameters, the quality of the training data is guaranteed, so that the training cost can be reduced while taking into account the model performance.

[0127] In an embodiment of the present disclosure, an electronic device is also provided, including:

[0128] Processor; and

[0129] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0130] In an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above-mentioned embodiments.

[0131] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements any of the above-mentioned embodiments of the method.

[0132] Figure 5 FIG. 5 shows a schematic diagram of the structure of an electronic device 5000. Figure 5 As shown, the electronic device 5000 includes: a processor 5010 and a memory 5020 , and optionally, may also include a power supply 5030 , a display unit 5040 , and an input unit 5050 .

[0133] The processor 5010 is the control center of the electronic device 5000. It connects various components using various interfaces and lines, and performs various functions of the electronic device 5000 by running or executing software programs and / or data stored in the memory 5020, thereby monitoring the electronic device 5000 as a whole.

[0134] In the embodiment of the present disclosure, the processor 5010 executes the various steps in the above embodiment when calling the computer program stored in the memory 5020.

[0135] Optionally, the processor 5010 may include one or more processing units; preferably, the processor 5010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 5010. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0136] The memory 5020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 5000, etc. In addition, the memory 5020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0137] The electronic device 5000 also includes a power source 5030 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 5010 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0138] The display unit 5040 may be used to display information input by the user or information provided to the user and various menus of the electronic device 5000, and in the embodiment of the present disclosure, is mainly used to display the display interface of each application in the electronic device 5000 and objects such as text and pictures displayed in the display interface. The display unit 5040 may include a display panel 5041. The display panel 5041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0139] The input unit 5050 may be used to receive information such as numbers or characters input by the user. The input unit 5050 may include a touch panel 5051 and other input devices 5052. The touch panel 5051, also known as a touch screen, may collect touch operations of the user on or near it (such as operations of the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 5051).

[0140] Specifically, the touch panel 5051 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 5010, and receive and execute the command sent by the processor 5010. In addition, the touch panel 5051 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 5052 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0141] Of course, the touch panel 5051 can cover the display panel 5041. When the touch panel 5051 detects a touch operation on or near it, it is transmitted to the processor 5010 to determine the type of touch event. Then the processor 5010 provides corresponding visual output on the display panel 5041 according to the type of touch event. Figure 5 In the embodiment, the touch panel 5051 and the display panel 5041 are used as two independent components to realize the input and output functions of the electronic device 5000, but in some embodiments, the touch panel 5051 and the display panel 5041 can be integrated to realize the input and output functions of the electronic device 5000.

[0142] The electronic device 5000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 5000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present disclosure, Figure 5 It is not shown and will not be described in detail.

[0143] Those skilled in the art will understand that Figure 5 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0144] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing the present disclosure, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

Claims

1. A method for training a network scheduling model, characterized in that: The method comprises: Generate offline training data according to the Lagrange multiplier, a plurality of network scheduling state samples and their corresponding network scheduling actions; the offline training data includes a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards; the reward is determined according to the corresponding network scheduling state samples and network scheduling actions, and the Lagrange multiplier; Adjusting parameters of the reinforcement learning model according to the offline training data; updating the Lagrange multiplier according to the adjusted reinforcement learning model; If it is determined that the updated Lagrange multiplier meets the convergence condition, the adjusted reinforcement learning model is used as the trained target network scheduling model; otherwise, the offline training data is updated according to the updated Lagrange multiplier, and the reinforcement learning model is adjusted based on the updated offline training data.

2. The method according to claim 1, characterized in that The step of adjusting parameters of the reinforcement learning model according to the offline training data includes: Determine the strategy target value according to multiple network scheduling state samples and their corresponding network scheduling actions and rewards; According to the strategy target value, the model parameters of the reinforcement learning model are preliminarily adjusted; According to the iteration step, the target strategy network is updated to obtain an updated target strategy network; The initially adjusted reinforcement learning model is adjusted again according to the updated targeted strategy network.

3. The method according to claim 2, characterized in that The determining of the strategy target value according to the plurality of network scheduling state samples and their corresponding network scheduling actions and rewards includes: Adopting an action value function, according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards, determining the action value function value corresponding to each network scheduling state sample; The strategy objective function is adopted to determine the strategy objective value according to multiple network scheduling state samples and their corresponding network scheduling actions, rewards and action value function values.

4. The method according to claim 3, characterized in that Before adopting the action value function to determine the action value function value corresponding to each network scheduling state sample according to the plurality of network scheduling state samples and their corresponding network scheduling actions and rewards, the method further includes: Adopting the action objective function, determining the action objective function value according to a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards; The action value function is updated according to the action objective function value.

5. The method according to any one of claims 1 to 4, characterized in that: The updating of the Lagrange multiplier according to the adjusted reinforcement learning model includes: Predicting network scheduling actions based on the collected network scheduling status and the adjusted reinforcement learning model; Determining network scheduling consumption energy according to the network scheduling state and its corresponding network scheduling action; The Lagrangian duality and energy constraint conditions are adopted to update the Lagrangian multiplier according to the energy consumed by the network scheduling.

6. The method according to any one of claims 1 to 4, characterized in that: The step of determining that the updated Lagrange multiplier meets the convergence condition includes: determining a difference between the Lagrangian multiplier before updating and the Lagrangian multiplier after updating; If the difference is lower than the convergence threshold, it is determined that the convergence condition is met.

7. A device for training a network scheduling model, characterized in that: The device comprises: A generating unit, configured to generate offline training data according to a Lagrange multiplier, a plurality of network scheduling state samples and their corresponding network scheduling actions; the offline training data includes a plurality of network scheduling state samples and their corresponding network scheduling actions and rewards; the rewards are determined according to the corresponding network scheduling state samples and network scheduling actions, and the Lagrange multiplier; An adjustment unit, used to adjust parameters of the reinforcement learning model according to the offline training data; An updating unit, used for updating the Lagrange multiplier according to the adjusted reinforcement learning model; A determination unit is used to use the adjusted reinforcement learning model as the trained target network scheduling model if it is determined that the updated Lagrange multiplier meets the convergence condition; otherwise, update the offline training data according to the updated Lagrange multiplier, and adjust the reinforcement learning model based on the updated offline training data.

8. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 6.

Images