Method and device for training neural network

By expanding the data obtained by interacting with the second agent and the environment into the training data of the first agent, the problem of slow convergence speed and easy entry into local optimal points in a non-stable environment is solved, and the convergence ability of reinforcement learning is improved and the accuracy of resource scheduling decisions.

CN112633491BActive Publication Date: 2024-12-03HUAWEI TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN201910951167.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-10-08
Publication Date
2024-12-03
Estimated Expiration
2039-10-08

AI Technical Summary

Technical Problem

Reinforcement learning converges slowly or does not converge in non-stable environments, and easily enters local optimality, especially in wireless resource scheduling tasks.

Method used

By expanding the data obtained by interacting with the second agent and the environment into the training data of the first agent, the stability and accuracy of the training data are improved, thereby improving the convergence ability of reinforcement learning.

Benefits of technology

It effectively improves the convergence speed of reinforcement learning algorithms in non-stable environments, avoids the problem of the training process entering local optimal points, and realizes more accurate agent training and resource scheduling decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112633491B_ABST
    Figure CN112633491B_ABST
Patent Text Reader

Abstract

The present application provides a method and device for training a neural network. It relates to the field of artificial intelligence, and specifically to the field of neural network training. The method includes: determining the training data of the first intelligent agent based on the first data obtained by the interaction between the first intelligent agent and the environment, and the second data obtained by the interaction between the second intelligent agent and the environment, the environment being the environment corresponding to the wireless resource scheduling task; and using the training data of the first intelligent agent to perform reinforcement learning training on the first intelligent agent. Since the training data of the first intelligent agent considers both the data of the interaction between the first intelligent agent and the environment and the data of the interaction between the second intelligent agent and the environment, it is beneficial to improve the stability and accuracy of the training data of the first intelligent agent. Therefore, the convergence ability of reinforcement learning training can be improved, which is beneficial to alleviate or avoid the problem that the training process of reinforcement learning enters the local optimum when it is applied to an unstable environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for training a neural network. Background Art

[0002] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Machine learning is the core of artificial intelligence. Machine learning methods include supervised learning and reinforcement learning.

[0003] The goal of supervised learning is to learn the mapping relationship between input and output in a training dataset given a training dataset. At the same time, it is hoped that the mapping relationship can also be applied to data outside the training dataset, that is, when new data arrives, the result can be predicted based on the mapping relationship. The training dataset is a collection of correct input-output pairs. Supervised learning requires obtaining labeled training datasets. Generally speaking, it is difficult to obtain labeled training data for decision problems.

[0004] Reinforcement learning was proposed for problems where it is difficult to obtain labeled training data sets, such as decision-making problems. Reinforcement learning is a learning method in which an agent learns by trial and error, and the reward obtained by interacting with the environment through actions guides the behavior, with the goal of maximizing the reward for the agent. Reinforcement learning is different from supervised learning, mainly in that it does not require a training data set.

[0005] Reinforcement learning learns through trial and error, and its convergence ability and convergence speed are much lower than supervised learning. In particular, when reinforcement learning is applied to tasks with unstable environments, such as wireless resource scheduling tasks in the communication field, the convergence speed of the reinforcement learning training process will be very slow or even non-convergent, for example, entering a local optimal point.

[0006] Therefore, improving the convergence ability of reinforcement learning is an urgent problem to be solved. Summary of the invention

[0007] The present application provides a method and device for training a neural network, which can improve the convergence ability of reinforcement learning.

[0008] In a first aspect, a method for training a neural network is provided, the method comprising: determining training data of a first intelligent agent based on first data obtained through interaction between a first intelligent agent and an environment, and second data obtained through interaction between a second intelligent agent and the environment, wherein the environment is an environment corresponding to a wireless resource scheduling task; and performing reinforcement learning training on the first intelligent agent using the training data of the first intelligent agent.

[0009] The first data represents data obtained by causing the first agent to interact with the environment, and the second data represents data obtained by causing the second agent to interact with the environment.

[0010] The first data includes the state and action obtained by the first agent interacting with the environment.

[0011] Optionally, the first data may also include performance indicators obtained by the interaction between the first agent and the environment.

[0012] Optionally, the first data may also include a reward corresponding to the state and action obtained by the interaction between the first agent and the environment. The reward may be obtained based on a performance indicator obtained by the interaction between the first agent and the environment.

[0013] The second data includes the state and action obtained by the interaction between the second agent and the environment.

[0014] Optionally, the second data may also include performance indicators obtained by the interaction between the second agent and the environment.

[0015] Optionally, the second data may also include a reward corresponding to the state and action obtained by the interaction between the second agent and the environment. The reward may be obtained based on a performance indicator obtained by the interaction between the second agent and the environment.

[0016] It should be understood that since the training data of the first agent not only takes into account the data of the interaction between the first agent and the environment, but also takes into account the data of the interaction between the second agent and the environment, this can improve the stability and accuracy of the training data of the first agent. Therefore, it can improve the convergence ability of the reinforcement learning training of the first agent to a certain extent, thereby improving the convergence speed.

[0017] Therefore, the solution provided in this application can improve the convergence ability of the reinforcement learning algorithm, thereby helping to alleviate or avoid the problem of the training process entering the local optimum when reinforcement learning is applied to an unstable environment.

[0018] In combination with the first aspect, in a possible implementation of the first aspect, the training data of the first intelligent agent is determined based on first data obtained through the interaction between the first intelligent agent and the environment, and second data obtained through the interaction between the second intelligent agent and the environment, including: taking the training data corresponding to the first data and the training data corresponding to the second data as the training data of the first intelligent agent.

[0019] The training data corresponding to the first data indicates the training data obtained from the first data. The training data corresponding to the second data indicates the training data obtained from the second data.

[0020] If the first data includes the state and action obtained by the interaction between the first intelligent agent and the environment, but does not include rewards, then the method for obtaining the training data corresponding to the first data is: obtaining the corresponding reward based on the performance indicator obtained by the interaction between the first intelligent agent and the environment; and obtaining the training data corresponding to the first data based on the obtained reward and the first data.

[0021] If the first data includes the state and action obtained by the interaction between the first agent and the environment, and also includes the corresponding reward, then the training data corresponding to the first data is the first data itself.

[0022] If the first data includes the state and action obtained by the interaction between the first intelligent agent and the environment, the corresponding reward, and the performance indicator obtained by the interaction between the first intelligent agent and the environment, then the training data corresponding to the first data is the state, action and reward included in the first data.

[0023] The above explanation of the training data corresponding to the first data is also applicable to the explanation of the training data corresponding to the second data, and for the sake of brevity, it will not be repeated here.

[0024] It should be understood that by expanding the training data of the second intelligent agent into the training data of the first intelligent agent, the richness of the training data of the first intelligent agent can be improved, thereby improving the stability and accuracy of the training data of the first intelligent agent. Therefore, the convergence ability of the reinforcement learning training of the first intelligent agent can be improved to a certain extent, thereby improving the convergence speed.

[0025] In addition, by expanding the training data of the second agent into the training data of the first agent, it can be regarded as the first agent's learning of the second agent, which is conducive to converting non-stationary problems into quasi-stationary problems, thereby helping to improve the convergence ability of the reinforcement learning algorithm.

[0026] In combination with the first aspect, in a possible implementation of the first aspect, the first data includes performance indicators obtained by the interaction between the first intelligent agent and the environment, and the second data includes performance indicators obtained by the interaction between the second intelligent agent and the environment; the reward in the training data of the first intelligent agent is obtained based on the performance indicators in the first data and the performance indicators in the second data.

[0027] As an optional implementation, the reward in the training data of the first agent is obtained by linearly weighting the difference between the performance indicator in the first data and the performance indicator in the second data.

[0028] As another optional implementation, the reward in the training data of the first agent is obtained by normalizing the difference between the performance indicator in the first data and the performance indicator in the second data.

[0029] It should be understood that determining the reward in the training data of the first intelligent agent based on the performance indicators obtained by the interaction between the first intelligent agent and the environment and the performance indicators obtained by the interaction between the second intelligent agent and the environment can improve the stability and accuracy of the reward in the training data of the first intelligent agent, and therefore, can improve the convergence ability of the reinforcement learning training of the first intelligent agent to a certain extent, thereby improving the convergence speed.

[0030] In addition, determining the reward in the training data of the first agent based on the performance indicators obtained by the interaction between the first agent and the environment and the performance indicators obtained by the interaction between the second agent and the environment can be regarded as establishing a basis for the performance indicators of the non-stationary environment, which is conducive to converting the non-stationary problem into a quasi-stationary problem, thereby helping to improve the convergence ability of the reinforcement learning algorithm.

[0031] In combination with the first aspect, in a possible implementation manner of the first aspect, a collection time of the second data is close to a collection time of the first data.

[0032] It should be understood that the collection time of the second data is close to the collection time of the first data, which can be beneficial to providing the accuracy of the training data of the first agent.

[0033] In combination with the first aspect, in a possible implementation of the first aspect, the method also includes: determining training data for the second intelligent agent based on the first data and the second data; and performing reinforcement learning training on the second intelligent agent using the training data for the second intelligent agent.

[0034] As an optional implementation method, the second agent is trained for reinforcement learning using the training data of the second agent, including: in all training rounds during the training process of the first agent, the second agent is trained for reinforcement learning using the training data of the second agent.

[0035] For example, the training of the second agent is performed alternately with the training of the first agent.

[0036] It should be understood that in this embodiment, by alternately allowing the first agent and the second agent to interact with the environment, and alternately allowing the first agent and the second agent to be trained, mutual learning between the first agent and the second agent can be achieved, which is beneficial for converting non-stationary problems into quasi-stationary problems, and is beneficial for improving the convergence ability of the reinforcement learning algorithm.

[0037] As another optional implementation, the second agent is trained with reinforcement learning using the training data of the second agent, including: in interval training rounds during the training of the first agent, the second agent is trained with reinforcement learning using the training data of the second agent.

[0038] It should be understood that in this embodiment, the computational complexity of the reinforcement learning training of the agents can be reduced by using the training data of the second agent to perform reinforcement learning training on the second agent in interval training rounds during the training of the first agent, rather than training the second agent in every round of training of the first agent.

[0039] In combination with the first aspect, in a possible implementation of the first aspect, the method further includes: obtaining a resource scheduling decision of the environment based on the trained first intelligent agent.

[0040] Optionally, the current state of the environment is used as the input of the trained first agent to obtain the action output by the first agent; and the current resource scheduling decision of the resource scheduling task is determined based on the action output by the first agent.

[0041] Optionally, the action output by the first agent may be used as the resource scheduling decision for this resource scheduling.

[0042] Optionally, according to a traditional scheduling decision method, a first scheduling decision of a wireless resource scheduling environment is obtained; and according to a second scheduling decision corresponding to the first scheduling decision and the action output by the first intelligent agent, a resource scheduling decision for this resource scheduling is determined.

[0043] In combination with the first aspect, in a possible implementation of the first aspect, the method further includes: obtaining a resource scheduling decision of the environment based on the trained first agent and the trained second agent.

[0044] Optionally, the current state of the wireless resource scheduling environment is used as the input of the trained first agent and the second agent, respectively, to obtain action 1 output by the first agent and action 2 output by the second agent. The resource scheduling decision for this resource scheduling is determined based on the scheduling decision corresponding to action 1 output by the first agent and the scheduling decision corresponding to action 2 output by the second agent.

[0045] Optionally, the resource scheduling decision for this resource scheduling is determined based on the scheduling decision corresponding to action 1 output by the first agent, the scheduling decision corresponding to action 2 output by the second agent, and the scheduling decision obtained by the traditional scheduling decision method.

[0046] It should be understood that the method provided in the present application is helpful in alleviating or avoiding the problem that the training process of reinforcement learning enters the local optimum when it is applied to an unstable environment. Therefore, a relatively accurate intelligent agent can be trained through the method provided in the present application, and thus a more reasonable resource scheduling strategy can be obtained based on the intelligent agent.

[0047] According to a second aspect, a communication device is provided, which can be used to execute the method in the first aspect.

[0048] Optionally, the communication device may include a module for executing the method in the first aspect.

[0049] Optionally, the communication device is a network device.

[0050] Optionally, the communication device is a chip.

[0051] For example, the communication device is a chip or circuit configured in a network device. For example, the communication device can be called an AI module.

[0052] In a third aspect, a communication device is provided, which includes a processor coupled to a memory, the memory being used to store computer programs or instructions, and the processor being used to execute the computer programs or instructions stored in the memory, so that the method in the first aspect is executed.

[0053] For example, the processor is used to execute the computer program or instructions stored in the memory, so that the communication device executes the method in the first aspect.

[0054] Optionally, the communication device includes one or more processors.

[0055] Optionally, the communication device may further include a memory coupled to the processor.

[0056] Optionally, the communication device may include one or more memories.

[0057] Optionally, the memory may be integrated with the processor, or provided separately.

[0058] Optionally, the communication device may also include a transceiver.

[0059] In a fourth aspect, a chip is provided, which includes a processing module and a communication interface, wherein the processing module is used to control the communication interface to communicate with the outside, and the processing module is also used to implement the method in the first aspect.

[0060] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program (also referred to as instructions or codes) for implementing the method in the first aspect is stored.

[0061] For example, when the computer program is executed by a computer, the computer can execute the method in the first aspect. The computer can be a communication device.

[0062] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program (also referred to as instructions or codes), wherein when the computer program is executed by a computer, the computer implements the method in the first aspect. The computer may be a communication device.

[0063] From the above, it can be seen that in the present application, since the training data of the first intelligent agent not only takes into account the data of the interaction between the first intelligent agent and the environment, but also takes into account the data of the interaction between the second intelligent agent and the environment, it is beneficial to improve the stability and accuracy of the training data of the first intelligent agent. Therefore, the convergence ability and convergence speed of the reinforcement learning training of the first intelligent agent can be improved to a certain extent, which is beneficial to alleviate or avoid the problem that when reinforcement learning is applied to an unstable environment, its training process enters the local optimal point. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a schematic diagram of the Markov decision process.

[0065] Figure 2 It is a schematic diagram of reinforcement learning.

[0066] Figure 3 It is a schematic diagram of a wireless resource scheduling scenario to which an embodiment of the present application can be applied.

[0067] Figure 4 It is a schematic flowchart of a method for training a neural network according to an embodiment of the present application.

[0068] Figure 5 is another schematic flowchart of a method for training a neural network according to an embodiment of the present application.

[0069] Figure 6 is another schematic flowchart of a method for training a neural network according to an embodiment of the present application.

[0070] Figure 7 It is a schematic block diagram of a device for training a neural network provided in an embodiment of the present application.

[0071] Figure 8 It is a schematic block diagram of an apparatus for training a neural network provided in another embodiment of the present application.

[0072] Fig. 9 It is a schematic block diagram of a communication device provided in an embodiment of the present application.

[0073] Fig.10 It is a schematic block diagram of a network device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0075] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in the field of artificial intelligence includes robotics, speech recognition, image recognition, natural language processing, decision-making and reasoning, human-computer interaction, recommendation and search, etc.

[0076] Machine learning is the core of artificial intelligence. Industry experts define machine learning as a process of gradually improving model performance P through training process E in order to achieve task T. For example, let a model recognize whether a picture is a cat or a dog (task T). In order to improve the accuracy of the model (model performance P), the model is continuously provided with pictures to learn the difference between cats and dogs (training process E). Through this learning process, the final model obtained is the product of machine learning. Ideally, the final model has the function of identifying cats and dogs in pictures. This training process is the learning process of machine learning.

[0077] Machine learning methods include supervised learning and reinforcement learning.

[0078] The goal of supervised learning is to learn the mapping relationship between input and output in a training data set given a training data set. At the same time, it is hoped that the mapping relationship can also be applied to data outside the training data set, that is, when new data arrives, the result can be predicted based on the mapping relationship. The training data set is a collection of correct input-output pairs. Usually, the correct input-output pairs in the training data set are labeled by humans. In other words, supervised learning can be regarded as providing a data set consisting of "correct answers" to the learning algorithm.

[0079] As we know, supervised learning requires a labeled training data set (i.e., a data set consisting of "correct answers"). However, for some tasks, it is difficult to obtain a labeled training data set. For example, for decision-making problems, it is difficult to obtain labeled training data.

[0080] Reinforcement learning was proposed for tasks where it is difficult to obtain labeled training datasets (e.g., decision-making problems).

[0081] Reinforcement learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals by learning strategies during its interaction with the environment.

[0082] A common model for reinforcement learning is the Markov decision process (MDP). MDP is a mathematical model for analyzing decision problems, such as Figure 1 , which assumes that the environment has a Markov property (the conditional probability distribution of the future state of the environment depends only on the current state), and the decision maker observes the state of the environment periodically (corresponding to Figure 1 s in the current environment), make decisions (also called actions, corresponding to Figure 1 Mark a in the figure), and obtain a new state after interacting with the environment (corresponding to Figure 1 The mark s) and reward (corresponding to Figure 1 The mark r in the .

[0083] Reinforcement learning is a learning method in which an agent learns by trial and error. The reward obtained by interacting with the environment through actions guides the agent's behavior, with the goal of maximizing the reward. Reinforcement learning is different from supervised learning, mainly because it does not require a training data set. In reinforcement learning, the reinforcement signal (i.e., reward) provided by the environment evaluates the quality of the action, rather than telling the reinforcement learning system how to generate the correct action. Since the external environment provides little information, the agent must learn through its own experience. In this way, the agent acquires knowledge in an action-evaluation (i.e., reward) environment and improves its action plan to adapt to the environment. Common reinforcement learning algorithms include Q-learning, policy gradient, actor-critic, etc.

[0084] like Figure 2 As shown in the figure, reinforcement learning mainly includes four elements: agent, environment state, action and reward, where the input of the agent is the state and the output is the action.

[0085] In current technology, the training process of reinforcement learning is: through multiple interactions between the agent and the environment, the action, state, and reward of each interaction are obtained; these multiple sets of (action, state, reward) are used as training data to train the agent once. Using the above process, the agent is trained for the next round until the convergence condition is met.

[0086] The process of obtaining the action, status, and reward of an interaction is as follows: Figure 2As shown in the figure, the current state s0 of the environment is input to the agent, and the action a0 output by the agent is obtained. The reward r0 of this interaction is calculated according to the relevant performance indicators of the environment under the action a0. At this point, the action a0, action a0 and reward r0 of this interaction are obtained. The action a0, action a0 and reward r0 of this interaction are recorded for subsequent training of the agent. The next state s1 of the environment under the action a0 is also recorded to realize the next interaction between the agent and the environment.

[0087] Reinforcement learning learns through trial and error, so its convergence ability and convergence speed are much lower than supervised learning.

[0088] In practical applications, the environment of some decision-making tasks is non-stationary, for example, wireless resource scheduling tasks in the communication field. When reinforcement learning is applied to wireless resource scheduling tasks, factors such as the user's location and changes in the wireless channel will affect the environment's feedback on actions (scheduling actions). For example, at different times, the same state and action correspond to different rewards, which will cause the convergence speed of the reinforcement learning training process to be very slow or even non-convergent, for example, entering a local optimal point.

[0089] The increasingly mature artificial intelligence technology will play an important role in promoting the evolution of future mobile communication network technology. At present, the academic community has conducted a lot of research on the application of artificial intelligence technology to the network layer (such as network optimization, mobility management, resource allocation, etc.) and the physical layer (such as channel coding, channel prediction, receiver, etc.). It can be foreseen that the application of reinforcement learning to wireless resource scheduling tasks is a future trend.

[0090] Therefore, improving the convergence ability of reinforcement learning is an urgent problem to be solved.

[0091] To address this problem, the present application provides a method and device for training a neural network, which can improve the convergence ability of reinforcement learning.

[0092] In the present application, when acquiring the training data of an intelligent agent, the interaction data of multiple intelligent agents and the environment are comprehensively considered, which can enhance the stability of the training data of the intelligent agent and thus reduce the adverse effects of an unstable environment on reinforcement learning.

[0093] The embodiments of the present application can be applied to various communication systems, for example, a long term evolution (LTE) system, a fifth generation mobile communication (5G) system, a machine to machine communication (M2M) system, or other communication systems that will evolve in the future. Among them, the wireless air interface technology of 5G is called new radio (NR), and the 5G system can also be called an NR system.

[0094] The terminal device involved in the embodiments of the present application may refer to user equipment (UE), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device. The terminal device may also be a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, a terminal device in a 5G network or a terminal device in a public land mobile communication network (PLMN) to be evolved in the future, etc.

[0095] The network device involved in the embodiments of the present application can be used to communicate with one or more terminals, and can also be used to communicate with one or more base stations having partial terminal functions (such as communication between a macro base station and a micro base station, such as an access point). The base station can be an evolved Node B (eNB) in an LTE system, or a base station (gNB) in a 5G system or NR system. In addition, the base station can also be an access point (AP), a transport node (TRP), a central unit (CU) or other network entities, and can include some or all of the functions of the above network entities.

[0096] Figure 3 Schematic diagram of a wireless resource scheduling scenario to which the embodiment of the present application can be applied. Figure 3 The application scenario shown includes a network device 310 and a terminal device 320, and the network device 310 can allocate scheduling resources for the terminal device 320. For example, the network device 310 receives information reported by the terminal device 320, such as a channel state. The network device 310 caches the user queue information sent by the upper layer and waits for it to be sent. In each transmission time interval (TTI), the network device 310 prioritizes users whose user queue information is not empty based on the information reported by the terminal device 320 and / or the information obtained by itself, and allocates resources to each user in order of priority and sends data.

[0097] The network device 310 may include a resource scheduling module for scheduling resources for users.

[0098] It should be understood that Figure 3 This is for example only and not limiting. For example, Figure 3 Four terminal devices 320 are schematically shown. In actual applications, the network device 310 can schedule resources for more terminal devices.

[0099] It should be noted that the reinforcement learning training method provided in this application can be applied to all tasks that are suitable for solving problems using reinforcement learning algorithms. For example, the task has a stable environment. Or, the task has an unstable environment, for example, the task is such as Figure 3 The wireless resource scheduling task shown.

[0100] For ease of description and understanding, the present application takes wireless resource scheduling as an example for description.

[0101] As described above, reinforcement learning involves Figure 2 The four elements shown are: agent, environment state, action, and reward. In this application, the application scenario of wireless resource scheduling is taken as an example to describe the training method of reinforcement learning. The following first defines the state of the environment (the environment corresponding to the wireless resource scheduling task) and the action of the agent in reinforcement learning when reinforcement learning is applied to the wireless resource scheduling task.

[0102] Define the state of the wireless resource scheduling environment as S = (E N×M ,V 1×M ,T 1×M ,B 1×M ). Where E N×M V represents the estimated transmission code rate of each user on each minimum scheduling unit (e.g., RBG), N represents the number of resources, and M represents the maximum number of users. 1×M represents the windowed average bit rate of each user. 1×M Indicates the waiting time of the longest waiting packet in each user cache. 1×M Indicates the size of each user's cache. N×M ,V 1×M ,T 1×M ,B 1×M All are real numbers.

[0103] Define the action of the agent in reinforcement learning as A N×M , A N×M Indicates the priority of user scheduling. N×M is a real number.

[0104] When reinforcement learning is applied to wireless resource scheduling tasks, the rewards in reinforcement learning can be determined based on performance indicators of the environment. For example, the performance indicators of the environment corresponding to the wireless resource scheduling task include but are not limited to: throughput, fairness, packet loss rate and other parameters.

[0105] Among them, throughput refers to the amount of data successfully transmitted per unit time for a network, device, port, virtual circuit or other facility. For example, throughput can be measured in units such as bits, bytes or packets.

[0106] Fairness refers to an indicator for measuring resource allocation among users in a system. For example, the commonly used fairness metric is Jain's fairness index.

[0107] The packet loss rate refers to the ratio of packets discarded due to overflow and timeout in the user cache to the total number of packets sent.

[0108] It should be understood that the above definitions of the states and actions of the wireless resource scheduling environment are merely examples and not limitations. In actual applications, the states and actions of the wireless resource scheduling environment may be defined according to application requirements.

[0109] The method for training a neural network provided in this application can be applied to network devices.

[0110] Optionally, the network device may be improved in software according to the method for training a neural network provided in the present application. For example, the reinforcement learning algorithm in the network device may be determined based on the method for training a neural network provided in the present application.

[0111] Optionally, the network device may be improved in hardware according to the method for training a neural network provided in the present application. For example, the device that can implement the method for training a neural network provided in the present application is used as a resource scheduling strategy module in the network device.

[0112] It should be noted that the training of the intelligent agent mentioned in the following embodiments refers to the training of the intelligent agent to perform reinforcement learning.

[0113] Figure 4 The method for training a neural network according to an embodiment of the present application is schematically shown in the flowchart. For example, the execution subject of the method may be a network device, or may be a resource scheduling module configured in the network device. The method includes the following steps.

[0114] S410, determining training data of the first agent according to first data obtained by the first agent interacting with the environment and second data obtained by the second agent interacting with the environment, wherein the environment is an environment corresponding to the wireless resource scheduling task.

[0115] The first data represents data obtained by causing the first agent to interact with the environment, and the second data represents data obtained by causing the second agent to interact with the environment.

[0116] It should be noted that, in the present application, multiple agents interact with the same environment. In this embodiment, the first agent and the second agent interact with the same environment.

[0117] For example, in step S410, each time the first agent interacts with the environment, the state, action, and performance index of this interaction are recorded. The performance index represents the performance index of the environment that is concerned when calculating the reward. Each time the second agent interacts with the environment, the state, action, and performance index of this interaction are recorded. The performance index represents the performance index of the environment that is concerned when calculating the reward.

[0118] The first data includes the state and action obtained by the first agent interacting with the environment.

[0119] Optionally, the first data may also include performance indicators obtained by the interaction between the first agent and the environment.

[0120] Optionally, the first data may also include a reward corresponding to the state and action obtained by the first agent interacting with the environment. The reward may be obtained based on the performance indicator obtained by the first agent interacting with the environment. The method for obtaining the reward will be described below and will not be described here for the time being.

[0121] The second data includes the state and action obtained by the second agent interacting with the environment.

[0122] Optionally, the second data may also include performance indicators obtained by the interaction between the second agent and the environment.

[0123] Optionally, the second data may also include a reward corresponding to the state and action obtained by the interaction between the second agent and the environment. The reward may be obtained based on a performance indicator obtained by the interaction between the second agent and the environment.

[0124] As we can see from the previous article, the training data in reinforcement learning includes three elements: state, action, and reward.

[0125] The training data of the first agent includes three elements: state, action, and reward.

[0126] The training data of the agent mentioned in the embodiments of the present application refers to the training data in reinforcement learning, which includes three elements: state, action, and reward. When the training data of the agent is mentioned again below, it will not be explained again.

[0127] Determining the training data of the first agent based on the first data and the second data means that the determination of the training data of the first agent not only considers the data of the interaction between the first agent and the environment, but also considers the data of the interaction between the second agent and the environment. Step S410 will be further described below.

[0128] S420, using the training data of the first intelligent agent, performing reinforcement learning training on the first intelligent agent.

[0129] It should be understood that since the training data of the first agent not only takes into account the data of the interaction between the first agent and the environment, but also takes into account the data of the interaction between the second agent and the environment, this can improve the stability and accuracy of the training data of the first agent. Therefore, it can improve the convergence ability of the reinforcement learning training of the first agent to a certain extent, thereby improving the convergence speed.

[0130] Therefore, the solution provided in this application can improve the convergence ability of the reinforcement learning algorithm, thereby helping to alleviate or avoid the problem of the training process entering the local optimum when reinforcement learning is applied to an unstable environment.

[0131] The second agent represents another agent different from the first agent.

[0132] For example, the second agent may differ from the first agent in any one or more of the following attributes: neural network structure, reinforcement learning algorithm.

[0133] The neural network structure includes but is not limited to: the number of neural network layers and the weight configuration of each neural network layer.

[0134] In the present application, the implementation methods for determining the training data of the first agent may include the following.

[0135] The first implementation method is to expand the training data of other agents into the training data of the first agent.

[0136] The second implementation method is to comprehensively consider the interaction data between the first agent and the environment and the interaction data between other agents and the environment to obtain the reward in the training data of the first agent.

[0137] The first implementation will be described below.

[0138] Optionally, in step S410, training data corresponding to the first data and training data corresponding to the second data are used as training data for the first agent.

[0139] The training data corresponding to the first data indicates the training data obtained from the first data. The training data corresponding to the second data indicates the training data obtained from the second data.

[0140] If the first data includes the state and action obtained by the interaction between the first intelligent agent and the environment, but does not include rewards, then the method for obtaining the training data corresponding to the first data is: obtaining the corresponding reward based on the performance indicator obtained by the interaction between the first intelligent agent and the environment; and obtaining the training data corresponding to the first data based on the obtained reward and the first data.

[0141] If the first data includes the state and action obtained by the interaction between the first agent and the environment, and also includes the corresponding reward, then the training data corresponding to the first data is the first data itself.

[0142] If the first data includes the state and action obtained by the interaction between the first intelligent agent and the environment, the corresponding reward, and the performance indicator obtained by the interaction between the first intelligent agent and the environment, then the training data corresponding to the first data is the state, action and reward included in the first data.

[0143] The above explanation of the training data corresponding to the first data is also applicable to the explanation of the training data corresponding to the second data, and for the sake of brevity, it will not be repeated here.

[0144] For example, by allowing the first agent to interact with the environment, training data 1{s 0 ,a 0 ,r 0 ;s 1 ,a 1 ,r 1 ;…;s n1 ,a n1 ,r n1}. Where n1 represents the number of times the first agent interacts with the environment in a round of training, and s i ,a i ,r i represents the state, action and reward in the i-th interaction between the first agent and the environment, i = 0, 1, ..., n1. By making the second agent interact with the environment, we obtain training data 2{s' 0 ,a' 0 ,r' 0 ;s' 1 ,a' 1 ,r' 1 ;…;s' n2 ,a' n2 ,r' n2}. Where n2 represents the number of times the second agent interacts with the environment in this round of training, and s' j ,a' j ,r' jrepresents the state, action and reward in the jth interaction between the second agent and the environment, j = 0, 1, ..., n2. The training data of the first agent includes training data 1 and training data 2. That is, the training data of the first agent is {s 0 ,a 0 ,r 0 ;s 1 ,a 1 ,r 1 ;…;s n1 ,a n1 ,r n1 ;s' 0 ,a' 0 ,r' 0 ;s' 1 ,a' 1 ,r' 1 ;…;s' n2 ,a' n2 ,r' n2}.

[0145] In the first implementation, the training data of the first agent includes not only the state, action, and reward obtained by the interaction between the first agent and the environment, but also the state, action, and reward obtained by the interaction between the second agent and the environment. It can be considered that in the first implementation, the training data of the second agent is expanded into the training data of the first agent.

[0146] It should be understood that by expanding the training data of the second intelligent agent into the training data of the first intelligent agent, the richness of the training data of the first intelligent agent can be improved, thereby improving the stability and accuracy of the training data of the first intelligent agent. Therefore, the convergence ability of the reinforcement learning training of the first intelligent agent can be improved to a certain extent, thereby improving the convergence speed.

[0147] In addition, by expanding the training data of the second agent into the training data of the first agent, it can be regarded as the first agent's learning of the second agent, which is conducive to converting non-stationary problems into quasi-stationary problems, thereby helping to improve the convergence ability of the reinforcement learning algorithm.

[0148] Therefore, the solution provided in this application can improve the convergence ability of the reinforcement learning algorithm, thereby helping to alleviate or avoid the problem of the training process entering the local optimum when reinforcement learning is applied to an unstable environment.

[0149] The second implementation will be described below.

[0150] Optionally, in Figure 4In the illustrated embodiment, the first data includes performance indicators obtained by the interaction between the first agent and the environment, and the second data includes performance indicators obtained by the interaction between the second agent and the environment. In step S410, based on the first data and the second data, the training data of the first agent is obtained, wherein the reward in the training data of the first agent is obtained according to the performance indicators in the first data and the performance indicators in the second data.

[0151] Optionally, the reward in the training data of the first agent is obtained by linearly weighting the difference between the performance indicator in the first data and the performance indicator in the second data.

[0152] For example, the reward in the training data of the first agent is calculated according to the following formula (1):

[0153] reward=αΔe 0 +βΔe 1 +γΔe 2 +...

[0154] Among them, Δe i represents the difference between the i-th performance index of the first agent and the i-th performance index of the second agent. 0 ,e 1 ,e 2 ,…} represent the performance indicators that the environment cares about, and {α, β, γ...} represent weighting coefficients.

[0155] It should be understood that the i-th performance index of the first agent refers to the i-th performance index of the environment after interacting with the first agent. The i-th performance index of the second agent refers to the i-th performance index of the environment after interacting with the second agent.

[0156] For example, the weighting coefficients α, β, γ, . . . may all be equal to 1.

[0157] Optionally, the reward in the training data of the first agent is obtained by normalizing the difference between the performance indicator in the first data and the performance indicator in the second data.

[0158] For example, the reward in the training data of the first agent is calculated according to the following formula (2): j :

[0159]

[0160] in, represents the normalization factor, K represents the number of agents, I represents the number of performance indicators of interest, K-1\j represents excluding j from (K-1), r jkirepresents the reward that the jth agent receives relative to the kth agent for the ith performance indicator. In this embodiment, the jth agent is the first agent, and K is equal to 2.

[0161] As an example, r jki Satisfies the following formula:

[0162]

[0163] For example, when K=I=2, α 0 =α 1 =1,β 0 =β 1 =0.5, formula (2) can be expressed using the following Table 1.

[0164] Table 1

[0165]

[0166] It should be understood that the above formula (1) and formula (2) are only examples and not limitations. In practical applications, other feasible methods can be adopted according to application requirements to obtain the reward in the training data of the first agent according to the performance indicators in the first data and the performance indicators in the second data.

[0167] In the second implementation method, the reward in the training data of the first intelligent agent is determined according to the performance indicators obtained by the interaction between the first intelligent agent and the environment and the performance indicators obtained by the interaction between the second intelligent agent and the environment. This can improve the stability and accuracy of the reward in the training data of the first intelligent agent. Therefore, the convergence ability of the reinforcement learning training of the first intelligent agent can be improved to a certain extent, thereby improving the convergence speed.

[0168] In addition, determining the reward in the training data of the first agent based on the performance indicators obtained by the interaction between the first agent and the environment and the performance indicators obtained by the interaction between the second agent and the environment can be regarded as establishing a basis for the performance indicators of the non-stationary environment, which is conducive to converting the non-stationary problem into a quasi-stationary problem, thereby helping to improve the convergence ability of the reinforcement learning algorithm.

[0169] Therefore, the solution provided in this application can improve the convergence ability of the reinforcement learning algorithm, thereby helping to alleviate or avoid the problem of the training process entering the local optimum when reinforcement learning is applied to an unstable environment.

[0170] Optionally, in the first implementation manner, the reward in the training data of the agent may adopt the method for obtaining the reward in the training data of the first agent in the second implementation manner.

[0171] Optionally, in the first implementation, the reward in the training data of the agent may also be obtained in other ways.

[0172] For example, the reward in the training data of the first agent is obtained based on the linear weighted sum of various performance indicators.

[0173] For example, the reward in the training data of the first agent can be obtained by the following formula:

[0174] reward=αe 0 +βe 1 +γe 2 +...

[0175] Among them, {e 0 ,e 1 ,e 2 ,…} represent the performance indicators that the environment cares about, and {α, β, γ...} represent weights.

[0176] Optionally, in the second implementation, the training data of other agents can also be expanded into the training data of the first agent. The rewards in the training data of other agents can be obtained by the method of obtaining the rewards in the training data of the first agent in the second implementation, or by other feasible methods.

[0177] Optionally, in step S410, the collection time of the second data is close to the collection time of the first data.

[0178] For example, in one training epoch, the first agent interacts with the environment as follows.

[0179] t, the first agent interacts with the environment to obtain state 1, action 1 and performance index 1;

[0180] t+1, the second agent interacts with the environment to obtain state 2, action 2 and performance indicators;

[0181] t+2, the first agent interacts with the environment to obtain state 3, action 3 and performance indicators;

[0182] t+3, the second agent interacts with the environment to obtain state 4, action 4 and performance indicators;

[0183] And so on.

[0184] Assume that the training data of the first agent includes (state 1, action 1, reward 1; state 3, action 3 and performance indicator 3; ...), where reward 1 can be obtained based on performance indicator 1 and performance indicator 2, and reward 3 can be obtained based on performance indicator 3 and performance indicator 4.

[0185] It should be understood that the collection time of the second data is close to the collection time of the first data, which can be beneficial to providing the accuracy of the training data of the first agent.

[0186] From the above, it can be seen that in the present application, since the training data of the first intelligent agent not only takes into account the data of the interaction between the first intelligent agent and the environment, but also takes into account the data of the interaction between the second intelligent agent and the environment, it is beneficial to improve the stability and accuracy of the training data of the first intelligent agent. Therefore, the convergence ability and convergence speed of the reinforcement learning training of the first intelligent agent can be improved to a certain extent, which is beneficial to alleviate or avoid the problem that when reinforcement learning is applied to an unstable environment, its training process enters the local optimal point.

[0187] In the prior art, in order to solve the problem that the training process of reinforcement learning enters the local optimum when it is applied to an unstable environment, it is proposed to model the environment in reinforcement learning as a hidden Markov model (HMM) or a partially observed Markov process (POMDP). In these schemes, the reinforcement learning algorithm needs to learn the hidden states in the environment, which usually increases the complexity of the algorithm.

[0188] In this application, the environment in reinforcement learning is modeled as a general Markov model (MDP), which can alleviate or avoid the problem that the training process of reinforcement learning enters the local optimum when it is applied to an unstable environment. Therefore, compared with the prior art, the solution provided by this application can reduce the complexity of the reinforcement learning algorithm.

[0189] It should be understood that the training process of the first agent generally includes multiple rounds of training, wherein the number of rounds of training is determined by the convergence condition. For example, when the first agent after the Nth round of training meets the convergence condition, the training process of the first agent ends. In this example, the training process of the first agent includes N rounds.

[0190] It should also be understood that the process of each round of training of the first agent includes: obtaining training data of the first agent; and training the first agent using the obtained training data.

[0191] It should also be understood that after each round of training, it is determined whether the convergence conditions are met. If so, the training process of the first agent ends, and if not, the next round of training continues.

[0192] The convergence conditions mentioned in this application include but are not limited to any one of the following conditions: the number of training rounds is greater than a threshold, the training time is greater than a threshold, the loss function is less than a threshold, and the test performance reaches a threshold.

[0193] In the present application, the above step S410 may be used to obtain the training data of the first agent in part or all of the training rounds of the first agent.

[0194] Optionally, in each round of training of the first agent, the above step S410 is used to obtain training data of the first agent.

[0195] Optionally, in some rounds of training of the first agent, the training data of the first agent is obtained by using the above step S410. For example, in the remaining rounds of training of the first agent, the training data of the first agent can be obtained by using a traditional method.

[0196] For example, in the first N1 trainings of the first agent, the training data of the first agent is obtained by using the above step S410. N1 is less than N, where N represents the number of training rounds of the first agent.

[0197] For another example, in an interval training round of the first agent, the above step S410 is used to obtain the training data of the first agent.

[0198] As an example, in the first round of training of the first intelligent agent, the above step S410 is used to obtain the training data of the first intelligent agent; in the second round of training of the first intelligent agent, the traditional method is used to obtain the training data of the first intelligent agent; in the third round of training of the first intelligent agent, the above step S410 is used to obtain the training data of the first intelligent agent; in the fourth round of training of the first intelligent agent, the traditional method is used to obtain the training data of the first intelligent agent; and so on, until the convergence condition is met.

[0199] The above mentioned traditional method for obtaining the training data of the first agent means that any existing or future technology evolution method can be used to obtain the training data of the first agent. Figure 1 The described method obtains training data for a first agent.

[0200] In the present application, when acquiring the training data of the first intelligent agent, the interaction data between the second intelligent agent and the environment is taken into consideration, wherein the second intelligent agent can be trained in advance before training the first intelligent agent, or can be trained during the training process of the first intelligent agent.

[0201] Optionally, in Figure 4 In the illustrated embodiment, the method further includes: determining training data for the second agent based on the first data and the second data; and performing reinforcement learning training on the second agent using the training data for the second agent.

[0202] Optionally, the second agent is trained for reinforcement learning using the training data of the second agent, including: in all training rounds during the training process of the first agent, the second agent is trained for reinforcement learning using the training data of the second agent.

[0203] For example, the training of the second agent is performed alternately with the training of the first agent.

[0204] It should be understood that in this embodiment, by alternately allowing the first agent and the second agent to interact with the environment, and alternately allowing the first agent and the second agent to be trained, mutual learning between the first agent and the second agent can be achieved, which is beneficial for converting non-stationary problems into quasi-stationary problems, and is beneficial for improving the convergence ability of the reinforcement learning algorithm.

[0205] Optionally, the second agent is trained for reinforcement learning using the training data of the second agent, including: in an interval training round during the training of the first agent, the second agent is trained for reinforcement learning using the training data of the second agent.

[0206] It should be understood that in this embodiment, the computational complexity of the reinforcement learning training of the agents can be reduced by using the training data of the second agent to perform reinforcement learning training on the second agent in interval training rounds during the training of the first agent, rather than training the second agent in every round of training of the first agent.

[0207] The convergence conditions for training the second agent may be the same as or different from the convergence conditions for training the first agent.

[0208] Regarding the training of the second agent, the training data of the second agent may be obtained by adopting the method of step S410 in the embodiment of the present application, or by adopting a traditional method.

[0209] In order to better understand the solution provided by this application, Figure 5 Give an example. Figure 5 The environment in may be an environment corresponding to the wireless resource scheduling task.

[0210] Explain first Figure 5 The elements shown in . The part surrounded by the dotted box marked "Training 1" represents the training process of the first agent, and the part surrounded by the dotted box marked "Training 2" represents the training process of the second agent. State 1 and action 1 represent the state and action obtained by the interaction between the first agent and the environment. State 2 and action 2 represent the state and action obtained by the interaction between the second agent and the environment. Performance indicator 1 represents the performance indicator obtained by the interaction between the first agent and the environment. Reward 1 represents the reward obtained according to performance indicator 1. Performance indicator 2 represents the performance indicator obtained by the interaction between the second agent and the environment. Reward 2 represents the reward obtained according to performance indicator 2. Next state 1 represents the next state of the environment after action 1 output by the first agent is applied to the environment. Next state 2 represents the next state of the environment after action 2 output by the second agent is applied to the environment.

[0211] In this example, the training process of reinforcement learning includes the following steps.

[0212] Step 1: Initialize the agent so that the input of the agent is the state and the output is the action.

[0213] The initialization method of the first agent is similar to that of the second agent. For the sake of brevity, the initialization of the agent is described below by taking the first agent as an example.

[0214] For example, the first agent includes a set of neural networks, which can be called a policy network. The input of the policy network is the state, and the output is the action. It should be understood that the policy network makes the input of the first agent the state, and the output is the action.

[0215] Optionally, the first agent may also include another set of neural networks, whose output is the estimated reward and whose input is the action, state or intermediate result.

[0216] It should be understood that, depending on the reinforcement learning algorithm adopted by the first agent, in addition to the policy network, the first agent may also include one or more other neural networks, which is not limited in this application.

[0217] It should also be understood that the description of the first agent in the above step 1 can be applied to the description of the second agent.

[0218] The structures and reinforcement learning algorithms of the first agent and the second agent may be the same or different. The structure of the agent includes but is not limited to: the number of groups of neural networks contained in the agent and the number of layers of each group of neural networks.

[0219] It should be noted that even if the structures of the first agent and the second agent are the same, the weights on the neural networks of the first agent and the second agent are not exactly the same.

[0220] In other words, the first agent and the second agent are not two completely identical agents.

[0221] In step 2, the first agent and the second agent are trained alternately until convergence conditions are reached.

[0222] Step 2 includes the following sub-steps.

[0223] Sub-step 1), the first agent and the second agent obtain the state of the wireless environment, and make scheduling actions for the environment according to their own strategy network. Statistics are collected on the performance indicators of interest (e.g., throughput, fairness, packet loss rate, etc.) and the next state of the environment, and the state, action, performance indicator and next state are recorded respectively. This step can be performed multiple times until the data obtained is sufficient to support one training.

[0224] Sub-step 2), calculating the reward for each interaction based on the performance indicators recorded in sub-step 1), and training the neural network of the first agent based on the data obtained from all interactions.

[0225] In sub-step 3), the first agent and the second agent obtain the state of the wireless environment and make scheduling actions for the environment according to their own strategy network. The performance indicators of interest (e.g., throughput, fairness, packet loss rate, etc.) and the next state of the environment are counted, and the state, action, performance indicator and next state are recorded respectively. This step can be performed multiple times until the data obtained is sufficient to support one training.

[0226] Sub-step 4), calculate the reward for each interaction according to the performance indicator parameters in sub-step 3), and train the neural network of the second agent based on the data obtained from all interactions.

[0227] Optionally, sub-step 2) and sub-step 4) are not performed simultaneously.

[0228] For example, in sub-step 2), the neural network parameters of the second agent are frozen, and in sub-step 4), the neural network parameters of the first agent are frozen.

[0229] Optionally, sub-step 2) and sub-step 4) can be performed simultaneously.

[0230] In this case, sub-step 1) and sub-step 3) may be the same step, or may be two steps executed in sequence.

[0231] For example, each round of training in step 2 includes the above sub-steps 1) to 4).

[0232] From the above, it can be seen that in the present application, since the training data of the first intelligent agent not only takes into account the data of the interaction between the first intelligent agent and the environment, but also takes into account the data of the interaction between the second intelligent agent and the environment, it is beneficial to improve the stability and accuracy of the training data of the first intelligent agent. Therefore, the convergence ability and convergence speed of the reinforcement learning training of the first intelligent agent can be improved to a certain extent, which is beneficial to alleviate or avoid the problem that when reinforcement learning is applied to an unstable environment, its training process enters the local optimal point.

[0233] It should be noted that, in the present application, other intelligent agents participating in the acquisition of training data of the first intelligent agent may include one intelligent agent or multiple intelligent agents, and the present application does not limit this.

[0234] For example, other agents participating in the acquisition of training data of the first agent may include a second agent and a third agent. The description of the second agent in this article may also be applicable to the third agent.

[0235] In this example, if the first implementation method is used to obtain the training data of the first agent, the training data of the second agent and the training data of the third agent can be expanded into the training data of the first agent.

[0236] In this example, if the second implementation method is used to obtain the training data of the first agent, the reward in the training data of the first agent can be obtained based on the performance indicators obtained by the first agent, the second agent and the third agent interacting with the environment respectively.

[0237] It should be understood that in this application, the solution described by taking the second agent participating in the acquisition of the training data of the first agent as an example, those skilled in the art can logically infer the solution in the case where the other agents participating in the acquisition of the training data of the first agent include more agents. In order to avoid redundancy, the solution in the case where the other agents participating in the acquisition of the training data of the first agent include more agents is not described in this article.

[0238] After the trained first agent is obtained, a resource scheduling strategy in a wireless resource scheduling environment can be obtained based on the first agent.

[0239] The intelligent agent mentioned in the following embodiments refers to the intelligent agent after training.

[0240] Alternatively, if Figure 6 As shown, in the embodiment of the present application, the method also includes the following steps.

[0241] S430, using the current state of the environment as the input of the trained first intelligent agent to obtain the output action of the first intelligent agent.

[0242] S440, determining the current resource scheduling decision of the resource scheduling task according to the action output by the first agent.

[0243] Optionally, in step S440, the action output by the first agent is used as the resource scheduling decision for this resource scheduling.

[0244] For example, the current state of the wireless resource scheduling environment is used as the input of the first agent, and the action output by the first agent is A = {m 0 , m 1 , m 2 , m 3 , m 4}. This action A={m 0 , m 1 , m 2 , m 3 , m 4} represents the scheduling priority of 5 terminal devices on 1 resource block. For example, the terminal device with the highest scheduling priority is selected for scheduling in this resource block.

[0245] As an example, according to the formula argmax(m 0 , m 1 , m 2 , m 3 , m 4 )Determine the terminal device with the highest scheduling priority.

[0246] Optionally, in step S440, a first scheduling decision of a wireless resource scheduling environment is obtained according to a traditional scheduling decision method; and a resource scheduling decision for this resource scheduling is determined according to a second scheduling decision corresponding to the first scheduling decision and an action output by the first agent.

[0247] For example, the resource scheduling decision for this resource scheduling is determined by compromising the first scheduling decision and the second scheduling decision.

[0248] As an example, the current state of the wireless resource scheduling environment is used as the input of the first agent, and the action output by the first agent is A = {m 0 , m 1 , m 2 , m 3 , m 4}. This action A={m 0 , m 1 , m 2 , m 3 , m 4} represents the scheduling priority of 5 terminal devices on 1 resource block. For the current wireless resource scheduling environment, the scheduling priority of 5 terminal devices obtained according to the traditional scheduling decision method is A'={p 0 , p 1 , p 2 , p 3 , p 4}. The final scheduled terminal device is determined according to the following formula:

[0249] argmax(m 0 +p 0 , m 1 +p 1 , m 2 +p 2 , m 3 +p 3 , m 4 +p 4 )

[0250] For example, p in the above formula i With m i(i=0,1,2,3,4) can all be scaled by coefficients to adjust their weights in the final scheduling results.

[0251] Optionally, in an embodiment where the training of the first agent also includes the training of the second agent, a resource scheduling strategy in a wireless resource scheduling environment may be acquired based on the trained first agent and the trained second agent.

[0252] Optionally, in step S430, the current state of the wireless resource scheduling environment is used as the input of the trained first agent and the second agent, respectively, to obtain action 1 output by the first agent and action 2 output by the second agent. In step S440, the resource scheduling decision for this resource scheduling is determined according to the scheduling decision corresponding to action 1 output by the first agent and the scheduling decision corresponding to action 2 output by the second agent.

[0253] Optionally, in step S440, the resource scheduling decision for this resource scheduling is determined based on the scheduling decision corresponding to action 1 output by the first agent, the scheduling decision corresponding to action 2 output by the second agent, and the scheduling decision obtained by the traditional scheduling decision method.

[0254] As an example, the current state of the wireless resource scheduling environment is used as the input of the first agent and the second agent respectively, and the action output by the first agent is A1={m 0 , m 1 , m 2 , m 3 , m 4}, the action output by the second agent is A2 = {q 0 ,q 1 ,q 2 ,q 3 ,q 4 Action A1 = {m 0 , m 1 , m 2 , m 3 , m 4} represents the scheduling priority of 5 terminal devices on 1 resource block. Action A2 = {q 0 ,q 1 ,q 2 ,q 3 ,q 4} also represents the scheduling priority of 5 terminal devices on 1 resource block. The final scheduled terminal device is determined according to the following formula:

[0255] argmax(m 0 +q 0 , m 1 +q 1 , m2 +q 2 , m 3 +q 3 , m 4 +q 4 )

[0256] For example, m in the above formula i With q i (i=0,1,2,3,4) can all be scaled by coefficients to adjust their weights in the final scheduling results.

[0257] As another example, the current state of the wireless resource scheduling environment is used as the input of the first agent and the second agent respectively, and the action output by the first agent is A1={m 0 , m 1 , m 2 , m 3 , m 4}, the action output by the second agent is A2 = {q 0 ,q 1 ,q 2 ,q 3 ,q 4 Action A1 = {m 0 , m 1 , m 2 , m 3 , m 4} represents the scheduling priority of 5 terminal devices on 1 resource block. Action A2 = {q 0 ,q 1 ,q 2 ,q 3 ,q 4} also represents the scheduling priority of 5 terminal devices on 1 resource block. For the current wireless resource scheduling environment, the scheduling priority of 5 terminal devices obtained according to the traditional scheduling decision method is A3 = {p 0 , p 1 , p 2 , p 3 , p 4}. The final scheduled terminal device is determined according to the following formula:

[0258] argmax(m 0 +p 0 +q 0 , m 1 +p 1 +q 1 , m 2 +p 2 +q 2 , m 3 +p 3 +q 3 , m4 +p 4 +q 4 )

[0259] For example, p in the above formula i 、m i With q i (i=0,1,2,3,4) can all be scaled by coefficients to adjust their weights in the final scheduling results.

[0260] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0261] The above describes the method embodiment provided by the present application, and the following describes the device embodiment provided by the present application. It should be understood that the description of the device embodiment corresponds to the description of the method embodiment, so the content not described in detail can be referred to the method embodiment above, and for the sake of brevity, it will not be repeated here.

[0262] The embodiment of the present application can divide the functional modules of the device for training a neural network according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other feasible division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0263] Figure 7 A schematic block diagram of a device 700 for training a neural network provided in the present application. The device 700 can be used to execute the method for training a neural network provided in the above embodiment. The device 700 can be a network device, or it can be a chip or circuit configured in a network device. Optionally, the device 700 is a chip or circuit configured in a network device, in which case the device 700 can be called an AI module.

[0264] like Figure 7 As shown, the device 700 includes a processing unit 710 and a training unit 720. The processing unit 710 is used to determine the training data of the first agent according to the first data obtained by the interaction between the first agent and the environment and the second data obtained by the interaction between the second agent and the environment, wherein the environment is an environment corresponding to the wireless resource scheduling task. The training unit 720 is used to perform reinforcement learning training on the first agent using the training data of the first agent.

[0265] Optionally, the processing unit 710 is used to use the training data corresponding to the first data and the training data corresponding to the second data as training data for the first agent.

[0266] Optionally, the first data includes a performance indicator obtained by the interaction between the first agent and the environment, and the second data includes a performance indicator obtained by the interaction between the second agent and the environment; the reward in the training data of the first agent is obtained according to the performance indicator in the first data and the performance indicator in the second data. That is, the processing unit 710 is used to obtain the reward in the training data of the first agent based on the performance indicator obtained by the interaction between the first agent and the environment and the performance indicator obtained by the interaction between the second agent and the environment.

[0267] For example, the reward in the training data of the first agent is obtained by linearly weighting the difference between the performance indicator in the first data and the performance indicator in the second data.

[0268] For another example, the reward in the training data of the first agent is obtained by normalizing the difference between the performance indicator in the first data and the performance indicator in the second data.

[0269] Optionally, the collection time of the second data is close to the collection time of the first data.

[0270] Optionally, the processing unit 710 is further used to determine training data of the second agent based on the first data and the second data; the training unit 720 is further used to perform reinforcement learning training on the second agent using the training data of the second agent.

[0271] Optionally, the processing unit 710 is used to perform reinforcement learning training on the second agent using the training data of the second agent in all training rounds during the training process of the first agent.

[0272] Optionally, the processing unit 710 is used to train the second agent using the training data of the second agent in reinforcement learning during an interval training round during the training of the first agent.

[0273] Optionally, the processing unit 710 is used to obtain a resource scheduling decision for a wireless resource scheduling task based on the trained first agent.

[0274] Optionally, the processing unit 710 is used to obtain a resource scheduling decision for a wireless resource scheduling task based on the trained first agent and the trained second agent.

[0275] Optionally, the apparatus 700 provided in the above embodiment is a chip or a circuit configured in a network device.

[0276] For example, in this case, the device 700 may be referred to as an AI module.

[0277] Optionally, the apparatus 700 provided in the above embodiment is a network device.

[0278] Alternatively, if Figure 8 As shown, the device 700 may also include a transceiver unit 730. The processing unit 710 is also used to obtain a resource scheduling strategy based on the trained agent (such as the first agent in the above embodiment) obtained by the training unit 720, and the transceiver unit 730 is used to send resource scheduling configuration information to the terminal device based on the resource scheduling strategy.

[0279] The processing unit 710 and the training unit 720 in the above embodiment can be implemented by a processor or a processor-related circuit. The transceiver unit 730 can be implemented by a transceiver or a transceiver-related circuit. The transceiver unit 730 can also be called a communication unit or a communication interface.

[0280] like Fig. 9 As shown, the embodiment of the present application further provides a communication device 900. The communication device 900 includes a processor 910, the processor 910 is coupled to a memory 920, the memory 920 is used to store computer programs or instructions, and the processor 910 is used to execute the computer program or instructions stored in the memory 920, so that the method in the above method embodiment is executed.

[0281] Alternatively, if Fig. 9 As shown, the communication device 900 may further include a memory 920 .

[0282] Alternatively, if Fig. 9 As shown, the communication device 900 may further include a transceiver 930, and the transceiver 930 is used for receiving and / or sending signals. For example, the processor 910 is used to control the transceiver 930 to receive and / or send signals.

[0283] For example, the processor 910 is used to implement the processing-related operations performed by the terminal device in the above method embodiment, and the transceiver 930 is used to implement the sending and receiving-related operations performed by the terminal device in the above method embodiment.

[0284] The communication device 900 is used to implement the method in the above method embodiment. For example, the processor 910 is used to implement the processing-related operations in the above method embodiment, and the transceiver 930 is used to implement the sending-receiving-related operations in the above method embodiment.

[0285] Optionally, the communication device 900 is a chip or a circuit configured in a network device.

[0286] For example, in this case, the communication device 900 may be referred to as an AI module.

[0287] Optionally, the communication device 900 provided in the above embodiment is a network device.

[0288] For example, in this case, the processor 810 is used to obtain a resource scheduling strategy based on a trained agent (such as the first agent in the above embodiment), and the transceiver 930 is used to send resource scheduling configuration information to the terminal device based on the resource scheduling strategy.

[0289] The embodiment of the present application further provides a communication device 1000, which can be a network device or a chip. The communication device 1000 can be used to execute the method in the above method embodiment.

[0290] When the communication device 1000 is a network device, for example, it is a base station. Fig.10 A simplified schematic diagram of a base station structure is shown. The base station includes a portion 1010 and a portion 1020. The portion 1010 is mainly used for receiving and transmitting radio frequency signals and converting radio frequency signals into baseband signals; the portion 1020 is mainly used for baseband processing, controlling the base station, etc. The portion 1010 can generally be referred to as a transceiver unit, a transceiver, a transceiver circuit, or a transceiver, etc. The portion 1020 is generally the control center of the base station, and can generally be referred to as a processing unit, which is used to control the base station to perform the processing operations on the network device side in the above method embodiment.

[0291] The transceiver unit of part 1010 may also be referred to as a transceiver or a transceiver, etc., and includes an antenna and a radio frequency circuit, wherein the radio frequency circuit is mainly used for radio frequency processing. Optionally, the device used to implement the receiving function in part 1010 may be regarded as a receiving unit, and the device used to implement the sending function may be regarded as a sending unit, that is, part 1010 includes a receiving unit and a sending unit. The receiving unit may also be referred to as a receiver, a receiver, or a receiving circuit, etc., and the sending unit may be referred to as a transmitter, a transmitter, or a transmitting circuit, etc.

[0292] Part 1020 may include one or more boards, each of which may include one or more processors and one or more memories. The processor is used to read and execute the program in the memory to realize the baseband processing function and the control of the base station. If there are multiple boards, each board can be interconnected to enhance the processing capability. As an optional implementation, multiple boards may share one or more processors, or multiple boards may share one or more memories, or multiple boards may share one or more processors at the same time.

[0293] For example, in one implementation, section 1020 is used to execute Figure 4 Step S410 and step S420 in, or, for performing Figure 5Steps S410 to S440 in the embodiment, and / or part 1020 are also used to perform the processing-related steps in the method of the above embodiment. The transceiver unit of part 1010 is used to perform the steps related to the transceiver operation in the method of the above embodiment, for example, the transceiver unit of part 1010 is used to send resource scheduling configuration information to the terminal device according to the resource scheduling strategy determined based on the output of the intelligent agent that has completed the training.

[0294] It should be understood that Fig.10 This is only an example and not a limitation. The network device including the transceiver unit and the processing unit may not rely on Fig.10 The structure shown.

[0295] When the communication device 1000 is a chip, the chip includes a transceiver unit and a processing unit, wherein the transceiver unit may be an input / output circuit or a communication interface; and the processing unit may be a processor or a microprocessor or an integrated circuit integrated on the chip.

[0296] The embodiment of the present application also provides a computer-readable storage medium on which computer instructions for implementing the method in the above method embodiment are stored.

[0297] For example, when the computer program is executed by a computer, the computer can implement the method in the above method embodiment.

[0298] The embodiment of the present application also provides a computer program product including instructions, which, when executed by a computer, enables the computer to implement the method in the above method embodiment.

[0299] The explanation of the relevant contents and beneficial effects in any of the communication devices provided above can refer to the corresponding method embodiments provided above, and will not be repeated here.

[0300] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0301] Optionally, the network device involved in the embodiment of the present application includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. Among them, the hardware layer may include hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system of the operating system layer can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer may include applications such as browsers, address books, word processing software, and instant messaging software.

[0302] The embodiments of the present application do not specifically limit the specific structure of the execution subject of the method provided in the embodiments of the present application, as long as it is possible to communicate according to the method provided in the embodiments of the present application by running a program that records the code of the method provided in the embodiments of the present application. For example, the execution subject of the method provided in the embodiments of the present application may be a terminal device or a network device, or a functional module in the terminal device or the network device that can call and execute a program.

[0303] Various aspects or features of the present application may be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used herein may encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, a computer-readable medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, or a magnetic tape, etc.), an optical disk (e.g., a compact disc (CD), a digital versatile disc (DVD), etc.), a smart card, and a flash memory device (e.g., an erasable programmable read-only memory (EPROM), a card, a stick, or a key drive, etc.).

[0304] The various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0305] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0306] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM). For example, RAM can be used as an external cache. By way of example and not limitation, RAM may include the following forms: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0307] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0308] It should also be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0309] Those of ordinary skill in the art will appreciate that the units and steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of this application.

[0310] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0311] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0312] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0313] In addition, each functional unit in each embodiment of the present application may be integrated into one unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0314] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a computer software product, which is stored in a storage medium. The computer software product includes several instructions, which are used to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium may include, but is not limited to: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0315] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training a neural network, characterized in that: include: Determine training data of the first agent according to first data obtained by the interaction between the first agent and the environment, and second data obtained by the interaction between the second agent and the environment, wherein the environment is an environment corresponding to the wireless resource scheduling task, the first data includes performance indicators obtained by the interaction between the first agent and the environment, and the second data includes performance indicators obtained by the interaction between the second agent and the environment, The reward in the training data of the first agent is obtained by linearly weighting or normalizing the difference between the performance index in the first data and the performance index in the second data; The first agent is trained using reinforcement learning using the training data of the first agent.

2. The method according to claim 1, characterized in that The determining of the training data of the first agent according to the first data obtained by the interaction between the first agent and the environment and the second data obtained by the interaction between the second agent and the environment comprises: The training data corresponding to the first data and the training data corresponding to the second data are used as the training data of the first agent.

3. The method according to claim 1, characterized in that The collection time of the second data is close to the collection time of the first data.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Determining training data for the second agent based on the first data and the second data; The second agent is trained using the training data of the second agent using reinforcement learning.

5. The method according to claim 4, characterized in that The using the training data of the second agent to perform reinforcement learning training on the second agent includes: In all training rounds during the training of the first agent, the second agent is trained by reinforcement learning using the training data of the second agent.

6. The method according to claim 4, characterized in that The using the training data of the second agent to perform reinforcement learning training on the second agent includes: In an interval training round during the training of the first agent, the second agent is trained by reinforcement learning using the training data of the second agent.

7. The method according to any one of claims 1 to 3, characterized in that The method further comprises: According to the trained first intelligent agent, a resource scheduling decision of the environment is obtained.

8. The method according to claim 4, characterized in that The method further comprises: A resource scheduling decision of the environment is obtained according to the trained first intelligent agent and the trained second intelligent agent.

9. A device for training a neural network, characterized in that: include: a processing unit, configured to determine training data of the first agent according to first data obtained by the first agent interacting with an environment and second data obtained by the second agent interacting with the environment, wherein the environment is an environment corresponding to a wireless resource scheduling task, the first data includes a performance indicator obtained by the first agent interacting with the environment, and the second data includes a performance indicator obtained by the second agent interacting with the environment, The reward in the training data of the first agent is obtained by linearly weighting or normalizing the difference between the performance index in the first data and the performance index in the second data; A training unit is used to perform reinforcement learning training on the first intelligent agent using the training data of the first intelligent agent.

10. The device according to claim 9, characterized in that The processing unit is used to use the training data corresponding to the first data and the training data corresponding to the second data as the training data of the first intelligent agent.

11. The device according to claim 9, characterized in that The collection time of the second data is close to the collection time of the first data.

12. The device according to any one of claims 9 to 11, characterized in that The processing unit is further used to determine the training data of the second agent according to the first data and the second data; The training unit is also used to perform reinforcement learning training on the second agent using the training data of the second agent.

13. The device according to claim 12, characterized in that The processing unit is used to: In all training rounds during the training of the first agent, the second agent is trained by reinforcement learning using the training data of the second agent.

14. The device according to claim 12, characterized in that The processing unit is used to: In an interval training round during the training of the first agent, the second agent is trained by reinforcement learning using the training data of the second agent.

15. The device according to any one of claims 9 to 11, characterized in that The processing unit is used to: According to the trained first intelligent agent, a resource scheduling decision of the wireless resource scheduling task is obtained.

16. The device according to claim 12, characterized in that The processing unit is used to: A resource scheduling decision of the wireless resource scheduling task is obtained according to the trained first intelligent agent and the trained second intelligent agent.

17. The device according to any one of claims 9 to 11, characterized in that The device is a network device.

18. A network device, characterized in that: Comprising a device as claimed in any one of claims 9 to 17.

19. A network device, characterized in that: include: A memory for storing executable instructions; A processor, configured to call and run the executable instructions in the memory to execute the method according to any one of claims 1 to 8.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

21. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Reinforcement learning multi-agent communication and decision method

    CN108921298A

  • A D2D resource allocation method based on multi-agent deep reinforcement learning

    CN109729528A

  • Multi-agent cooperation framework based on deep reinforcement learning

    CN110084375A

  • Method and device applied to reinforcement learning training process and electronic equipment

    CN110147891A

  • Joint resource allocation and power control method in D2D communication

    CN110267338A