Learning device, communication device, unmanned aerial vehicle, wireless communication system, learning method, and learning program

Abstract

To provide technology for establishing an unmanned vehicle system appropriately and achieving performance improvement that is difficult to achieve only with conventional wireless communication designs.SOLUTION: A learning device includes: a setting unit that sets a first requirement value for a predetermined parameter of a communication device controlled by a computer using a learned model; a reinforcement learning unit that allows a learning model to learn; a model extraction unit that extracts, as a learned model, the learning model in which a number of learning steps is equal to or greater than a predetermined number; a model evaluation unit that determines whether or not performance of the extracted learned model has reached first requirement performance; an updating unit that updates the first requirement value to a second requirement value when the performance of the learned model is determined to have reached the first requirement performance; and a model selection unit that selects the learned model to be installed in the computer. When the performance of the learned model updated to the second requirement value is determined to satisfy second requirement performance, the model selection unit selects the learned model that satisfies the second requirement performance as the learned model to be installed in the computer.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a communication device, an unmanned aerial vehicle, a wireless communication system, a learning method, and a learning program. [Background technology]

[0002] There are known techniques for managing unmanned aerial vehicles. For example, Patent Document 1 discloses a technique for broadcasting geolocation information from an unmanned aerial vehicle to inform others of the current geolocation (geographical position) of the unmanned aerial vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6853897 Summary of the Invention [Problem to be solved by the invention]

[0004] In an unmanned aircraft system with multiple unmanned aircraft, in order for each unmanned aircraft to act in coordination, information about each unmanned aircraft is needed in real time, in addition to the strategies of other unmanned aircraft obtained through reinforcement learning, so it is desirable to transmit information at a high rate. For this reason, in unmanned aircraft systems, for example, a TDMA (Time Division Multiple Access) network is formed to prevent collisions between the transmitted radio waves of each unmanned aircraft, and slots for transmitting information are set according to the number of unmanned aircraft included in the unmanned aircraft system.

[0005] When an unmanned aircraft system includes a large number of unmanned aircraft, such as tens or hundreds, each unmanned aircraft requires a slot to occupy, and the overhead of reserving slots can result in reduced throughput. Possible solutions to this throughput reduction include increasing the wireless bandwidth, increasing the transmission power, and improving the receiver sensitivity. However, increasing the wireless bandwidth may result in interference with communications with other stations, making it difficult to obtain permission for use. Furthermore, increasing the transmission power, for example, increases the weight of the wireless communication device, which is detrimental to miniaturization. As a result, it may be difficult to establish an unmanned aircraft system.

[0006] Therefore, the objective of the present disclosure is to provide a learning device, communication device, unmanned aircraft, wireless communication system, learning method, and learning program that can properly establish an unmanned aircraft system by applying reinforcement learning to the design of wireless communication, and further improve performance that is difficult to achieve with conventional wireless communication design alone. [Means for solving the problem]

[0007] The learning device of the present disclosure is a learning device for training a trained model installed in a computer, and includes a setting unit that sets a first required value for a predetermined parameter of a communication device controlled by the computer using the trained model; a reinforcement learning unit that trains the trained model so as to maximize a reward to be given in a predetermined environment; a model extraction unit that extracts the trained model with a predetermined number of learning steps or more as a trained model; a model evaluation unit that determines whether the performance of the trained model extracted by the model extraction unit has reached a first required performance; and an updating unit that updates the first required value to a second required value different from the first required value when it determines that the first required value has been reached, and a model selecting unit that selects the trained model to be installed in the computer, wherein the model evaluation unit determines whether the performance of the trained model updated to the second required value satisfies the second required performance different from the first required performance, and when the model evaluation unit determines that the performance of the trained model updated to the second required value satisfies the second required performance, the model selecting unit selects the trained model that satisfies the second required performance as the trained model to be installed in the computer.

[0008] The communication device of the present disclosure performs communication based on control using a trained model trained by the learning device of the present disclosure.

[0009] The unmanned aircraft of the present disclosure comprises a computer equipped with a trained model trained by the learning device of the present disclosure, and a communication device, and the computer communicates via the communication device using the trained model.

[0010] The wireless communication system of the present disclosure includes a plurality of unmanned aerial vehicles of the present disclosure.

[0011] The learning method disclosed herein is a learning method for training a trained model to be installed in a computer, and includes the steps of: setting a first required value for a predetermined parameter of a communication device controlled by the computer using the trained model; training the trained model so as to maximize a reward to be awarded in a predetermined environment; extracting the trained model for which the number of learning steps is equal to or greater than a predetermined number as a trained model; determining whether the performance of the extracted trained model has reached a first required performance; if it is determined that the performance of the trained model has reached the first required performance, updating the first required value to a second required value different from the first required value; determining whether the performance of the trained model updated to the second required value satisfies the second required performance different from the first required performance; and, if it is determined that the performance of the trained model updated to the second required value satisfies the second required performance, selecting the trained model that satisfies the second required performance as the trained model to be installed in the computer.

[0012] The learning program disclosed herein is a learning program for using a learning device to learn a trained model installed in a computer, and causes the learning device to execute the following steps: setting a first required value for a predetermined parameter of a communication device controlled by the computer using the trained model; training the trained model so as to maximize the reward awarded in a predetermined environment; extracting the trained model with a predetermined number of learning steps or more as a trained model; determining whether the performance of the extracted trained model has reached a first required performance; if it is determined that the performance of the trained model has reached the first required performance, updating the first required value to a second required value different from the first required value; determining whether the performance of the trained model updated to the second required value satisfies the second required performance different from the first required performance; and, if it is determined that the performance of the trained model updated to the second required value satisfies the second required performance, selecting the trained model that satisfies the second required performance as the trained model to be installed in the computer. [Effects of the Invention]

[0013] According to the present disclosure, an unmanned aircraft system can be properly established, and performance improvements that are difficult to achieve with conventional wireless communication designs alone can be achieved. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is an explanatory diagram regarding learning using a learning model according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of the flow of the learning process according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of an unmanned aircraft system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical. Furthermore, the components described below can be combined as appropriate, and when there are multiple embodiments, the respective embodiments can also be combined.

[0016] [Embodiment] The learning device 10 and learning method according to this embodiment are a device and method for learning a learning model including hyperparameters. Fig. 1 is an explanatory diagram of learning using the learning model according to this embodiment. Fig. 2 is a block diagram showing an example of the configuration of the learning device according to this embodiment.

[0017] (Learning using learning models) First, learning using a learning model M will be described with reference to Fig. 1. The learning model M is installed in an agent 2 that executes an action At. The agent 2 may be, for example, a machine capable of executing an action, such as a robot, a vehicle, a ship, or an aircraft. The agent 2 executes a predetermined action At in a predetermined environment 4 using the learning model M.

[0018] As shown in FIG. 1, the learning model M is a neural network having multiple nodes. The neural network is a network in which multiple nodes are connected, has multiple layers, and each layer has multiple nodes. Parameters of the neural network include weights and biases between nodes. Other parameters of the neural network include hyperparameters such as the number of layers, the number of nodes, and a learning rate. In this embodiment, the weights and biases between nodes of the learning model M are learned.

[0019] Next, learning using the learning model M will be explained. There are two types of learning: imitation learning and reinforcement learning. Imitation learning is supervised learning, in which the hyperparameters of the learning model M are learned so that the agent 2 executes a predetermined action At when a predetermined state St is input in a predetermined environment 4. Reinforcement learning is unsupervised learning, in which the hyperparameters of the learning model M are learned so that the reward Rt given to the agent 2 in the predetermined environment 4 is maximized.

[0020] In reinforcement learning, agent 2 acquires state St from environment 4, and also acquires reward Rt from environment 4. Agent 2 then selects action At from learning model M based on the acquired state St and reward Rt. When agent 2 executes the selected action At, state St of agent 2 transitions to state St+1 in environment 4. Agent 2 is also given a reward Rt+1 based on the executed action At, the state St before the transition, and the state St+1 after the transition. Then, in reinforcement learning, the above learning is repeated a predetermined number of evaluable steps so that the reward Rt given to agent 2 is maximized.

[0021] In the present disclosure, a learning device 10 learns a learning model to be installed in a computer in order to appropriately set the required performance of a communication device of an unmanned aerial vehicle. The learning device 10 determines an information transfer rate for the communication device of the unmanned aerial vehicle and performs reinforcement learning for the determined information transfer rate. If the learning device 10 determines that the determined information transfer rate satisfies a first required performance, the learning device 10 slows down the information transfer rate and performs reinforcement learning to satisfy a second required performance.

[0022] In the present disclosure, the unmanned aerial vehicle is not particularly limited, and may be, for example, any of an aircraft, a ship, and a vehicle.

[0023] (Learning device) Returning to Fig. 2, as shown in Fig. 2, the learning device 10 includes an environment unit 12, a storage unit 14, and a control unit 16.

[0024] The environment unit 12 provides an environment for executing reinforcement learning on the trained model. The environment unit 12 has an exercise model 20, an environment model 22, and a reward model 24. The environment unit 12 provides an environment for executing reinforcement learning based on the exercise model 20, the environment model 22, and the reward model 24. Specifically, the environment unit 12 provides rewards for the trained model and derives the state of the trained model that transitions depending on actions.

[0025] The storage unit 14 is a memory that stores various types of information. The storage unit 14 stores, for example, information such as the calculation contents of the control unit 16 and programs. The storage unit 14 includes, for example, at least one of a RAM (Random Access Memory), a main storage device such as a ROM (Read Only Memory), and an external storage device such as an HDD (Hard Disk Drive). The storage unit 14 stores a reinforcement learning model 30.

[0026] The reinforcement learning model 30 includes a plurality of trained models in reinforcement learning. The reinforcement learning model 30 stores, for example, a plurality of trained models trained for each learning step.

[0027] The control unit 16 controls the operation of each unit of the learning device 10. The control unit 16 is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing a program stored in the storage unit 14 or the like using RAM or the like as a work area. The control unit 16 may be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 16 may also be realized by a combination of hardware and software.

[0028] The control unit 16 includes a setting unit 40 and a learning unit 42.

[0029] The setting unit 40 sets various conditions for executing reinforcement learning. For example, the setting unit 40 sets a first requirement value for a predetermined parameter of the communication device. For example, the setting unit 40 sets a behavior judgment model (state and behavior), a reward function, a deep reinforcement learning algorithm, model granularity, hyperparameters, etc. In other words, the setting unit 40 creates an environment for executing reinforcement learning. Details of the setting unit 40 will be described later.

[0030] The learning unit 42 executes learning of the learning model and includes a reinforcement learning unit 50, a model extraction unit 52, a model evaluation unit 54, an update unit 56, and a model selection unit 58.

[0031] The reinforcement learning unit 50 performs learning based on the reward given by the environment unit 12. Details of the reinforcement learning unit 50 will be described later.

[0032] The model extraction unit 52 extracts, as a trained model, a learning model having a predetermined number of learning steps or more in reinforcement learning by the reinforcement learning unit 50. Details of the model extraction unit 52 will be described later.

[0033] The model evaluation unit 54 evaluates the trained model. For example, the model evaluation unit 54 determines whether the performance of the trained model extracted by the model extraction unit 52 reaches the first required performance. Details of the model evaluation unit 54 will be described later.

[0034] The update unit 56 updates a required value set for a predetermined parameter of the communication device. For example, when the model evaluation unit 54 determines that the performance of the trained model has reached the first required performance, the update unit 56 updates the first required value to a second required value different from the first required value. Details of the update unit 56 will be described later.

[0035] The model selection unit 58 selects a trained model to be installed in the communication device. Details of the model selection unit 58 will be described later.

[0036] (Learning process) The flow of the learning process according to this embodiment will be described using Fig. 3. Fig. 3 is a flowchart showing an example of the flow of the learning process according to this embodiment. Below, a process of learning the transmission speed of the information transfer rate of a communication device mounted on an unmanned aerial vehicle will be described as an example, but the present disclosure is not limited to this.

[0037] The setting unit 40 sets the information transfer rate of the communication device of the unmanned aerial vehicle to a first transmission speed (step S10). Specifically, the setting unit 40 sets the first transmission speed, which can be said to be a sufficiently high rate as the information transfer rate to be set in the communication device. The first transmission speed is a type of first required value. Then, the process proceeds to step S12.

[0038] The setting unit 40 creates various environments for executing reinforcement learning (step S12). Specifically, the setting unit 40 creates, for example, a simulation environment for executing reinforcement learning. Then, the process proceeds to step S14.

[0039] The setting unit 40 considers a learning model for executing reinforcement learning (step S14). Specifically, the setting unit 40 considers various aspects required for implementing reinforcement learning, such as a behavioral judgment model (state and behavior), a reward function, a deep reinforcement learning algorithm, model granularity, and hyperparameters. Then, the process proceeds to step S16.

[0040] The reinforcement learning unit 50 executes reinforcement learning (step S16). Specifically, the reinforcement learning unit 50 executes learning so as to maximize the reward given to the trained model, for example. Then, the process proceeds to step S18.

[0041] The reinforcement learning unit 50 determines whether the number of steps for which reinforcement learning has been performed is equal to or greater than a predetermined number of steps (step S18). The predetermined number of steps may be set arbitrarily depending on the problem being addressed, etc. If it is determined that the number of steps is equal to or greater than the predetermined number of steps (step S18; Yes), the process proceeds to step S20. If it is determined that the number of steps is not equal to or greater than the predetermined number of steps (step S18; No), the process proceeds to step S16. That is, in this embodiment, reinforcement learning is repeated until the predetermined number of steps is reached.

[0042] If the determination in step S18 is Yes, the model extraction unit 52 extracts trained models (step S20). Specifically, the model extraction unit 52 extracts all trained models that have undergone reinforcement learning for a predetermined number of steps or more. Then, the process proceeds to step S22.

[0043] The reinforcement learning unit 50 determines whether the number of steps for which reinforcement learning has been performed has reached the maximum number of steps (step S22). The maximum number of steps may be set arbitrarily. If it is determined that the number of steps has reached the maximum number of steps (step S22; Yes), the process proceeds to step S24. If it is determined that the number of steps has not reached the maximum number of steps (step S22; No), the process proceeds to step S16. That is, in this embodiment, reinforcement learning is repeated until the number of learning steps reaches the maximum number of steps.

[0044] If the determination in step S22 is Yes, the model evaluation unit 54 evaluates the performance of the trained model whose number of steps has reached the maximum number of steps (step S24), and then proceeds to step S26.

[0045] The model evaluation unit 54 determines whether the performance of the trained model has reached the first required performance (step S26). Specifically, the model evaluation unit 54 determines whether the trained model set to the first transmission rate set in step S10 has reached the desired first required performance. If it is determined that the performance of the trained model has reached the first required performance (step S26; Yes), the process proceeds to step S28. If it is not determined that the performance of the trained model has reached the first required performance (step S26; No), the process proceeds to step S16. That is, in this embodiment, reinforcement learning is repeated until it is determined that the performance of the trained model has reached the first required performance.

[0046] The update unit 56 updates the first transmission rate set in step S10 to a second transmission rate (step S28). Specifically, the update unit 56 updates the first transmission rate to a second transmission rate that is lower than the first transmission rate. The second transmission rate is a type of second required value. Then, the process proceeds to step S30.

[0047] The model evaluation unit 54 determines whether the trained model updated from the first transmission rate to the second transmission rate satisfies the desired second required performance (step S30). The second required performance may be the same as the first required performance, or may be inferior to the first required performance. The second required performance may be changed arbitrarily depending on the design. For example, the second required performance may be changed depending on the functions added to the communication device of the drone. If it is determined that the trained model updated to the second transmission rate satisfies the desired second required performance (step S30; Yes), the process proceeds to step S32. If it is not determined that the trained model updated to the second transmission rate satisfies the desired second required performance (step S30; No), the process proceeds to step S16. That is, in this embodiment, reinforcement learning is repeated until it is determined that the performance of the trained model satisfies the second required performance.

[0048] If the determination in step S30 is Yes, the model selection unit 58 selects a trained model (step S32). Specifically, the model selection unit 58 selects the trained model determined in step S30 to satisfy the second required performance as the trained model to be installed in the computer of the unmanned vehicle. Then, the processing in FIG. 3 ends.

[0049] In this embodiment, by applying the trained model selected by the learning device 10 to the design of an unmanned aircraft equipped with a communication device, it is possible to apply an unmanned aircraft system with a lower data rate than conventional systems. Specifically, this embodiment can realize an unmanned aircraft system in which each unmanned aircraft can operate in cooperation even in a low-rate communication environment. That is, in this embodiment, by learning in a high-rate state and then learning at a low rate, it is possible to appropriately set the data rate required by the unmanned aircraft system.

[0050] [Unmanned Aerial Systems] An example of the configuration of an unmanned aircraft system according to an embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of an unmanned aircraft system according to an embodiment.

[0051] As shown in Figure 4, unmanned aircraft system 100 includes unmanned aircraft 110, unmanned aircraft 112, and unmanned aircraft 114. Unmanned aircraft 110 to 114 may be, for example, but are not limited to, vehicles, ships, aircraft, etc. In the example shown in Figure 4, unmanned aircraft system 100 is described as including three unmanned aircraft, although the present disclosure is not limited thereto.

[0052] Unmanned aerial vehicle 110 has a computer 120 and a communication device 130. Unmanned aerial vehicle 112 has a computer 122 and a communication device 132. Unmanned aerial vehicle 114 has a computer 124 and a communication device 134. Communication device 130, communication device 132, and communication device 134 are communicatively connected via a wireless network N. Network N is, for example, a TDMA network, but is not limited to this.

[0053] The computers 120 to 124 are each equipped with a trained model that satisfies the second required performance, selected by the learning device 10 through the process shown in FIG. 3 . The computers 120 to 124 each include a processing device such as a CPU and a storage device such as a RAM or ROM. The computers 120 to 124 exchange information with other unmanned aircraft via the communication devices 130 to 134, using the trained model that satisfies the second required performance. That is, in this embodiment, the computers 120 to 124 are each equipped with a trained model, and communication is performed via the communication device using the trained model, thereby improving the feasibility of the unmanned aircraft system despite the low communication rate and further improving performance that is difficult to achieve with conventional wireless communication design alone.

[0054] Unmanned aerial vehicles 110 to 114 transmit and receive information about each drone using computers 120 to 124 and communication devices 130 to 134, respectively. In this embodiment, computers 120 to 124 use the trained model selected by learning device 10 to exchange information with other drones via the communication devices, thereby transmitting and receiving information about each drone at a low rate, thereby making it possible to obtain surplus resources for the wireless communication line. In other words, by obtaining surplus resources for the wireless communication line, unmanned aerial vehicle system 100 of this embodiment can extend communication distances, ensure future functional expandability, improve encryption strength, reduce the weight and power consumption of communication devices, and expand functions such as preventing interference with other stations.

[0055] The learning device, communication device, unmanned aerial vehicle, wireless communication system, learning method, and learning program described in this embodiment can be understood, for example, as follows.

[0056] The learning device of the first aspect is a learning device for training a trained model to be installed in a communication device, and includes: a setting unit 40 that sets a first requirement value for a predetermined parameter of the communication device controlled by a computer using the trained model; a reinforcement learning unit 50 that trains the trained model so as to maximize a reward to be awarded in a predetermined environment; a model extraction unit 52 that extracts trained models with a predetermined number of learning steps or more as trained models; a model evaluation unit 54 that determines whether the performance of the trained model extracted by the model extraction unit 52 has reached the first required performance; an update unit 56 that updates the first requirement value to a second requirement value different from the first required value when the model evaluation unit 54 determines that the performance of the trained model has reached the first required performance; and a model selection unit 58 that selects a trained model to be installed in the computer. The model evaluation unit 54 determines whether the performance of the trained model updated to the second requirement value satisfies the second required performance different from the first required performance. When the model evaluation unit 54 determines that the performance of the trained model updated to the second required value satisfies the second required performance, the model selection unit 58 selects the trained model that satisfies the second required performance as the trained model to be installed in the computer. In this way, the learning device of the first aspect appropriately selects a trained model that satisfies the required performance that can establish an unmanned aircraft system, and further enables performance improvement that is difficult to achieve with conventional wireless communication design alone.

[0057] In a second aspect, the update unit 56 changes the second required value in accordance with the second required performance, thereby allowing the learning device of the second aspect to arbitrarily set the required performance that can establish the unmanned aircraft system.

[0058] In a third aspect, the predetermined parameter of the communication device is an information transfer rate. The setting unit sets a first transmission rate as a first required value for the information transfer rate. The updating unit updates the first transmission rate to a second transmission rate that is slower than the first transmission rate. This allows the learning device of the third aspect to appropriately select a trained model that satisfies the required performance even when the information transfer rate is slow.

[0059] In a fourth aspect, a communication device performs communication based on control using a trained model trained by the learning device according to any one of the first to third aspects. This allows the communication device to secure surplus resources in the wireless communication line by slowing down the information transfer rate.

[0060] The unmanned aerial vehicle of the fifth aspect includes a computer 120 equipped with a trained model trained by the learning device of any one of the first to third aspects, and a communication device 130. The computer 120 uses the trained model to communicate via the communication device 130. This allows the communication device to secure surplus resources for the wireless communication line by slowing down the information transfer rate. The unmanned aerial vehicle of the fourth aspect can then transmit its own information to other vehicles at a slower rate.

[0061] The wireless communication system of the sixth aspect includes a plurality of unmanned aerial vehicles 110 of the fifth aspect. This allows the unmanned aerial vehicle system of the sixth aspect to operate in coordination with each other even in a communication environment with a low information transfer rate.

[0062] A learning method of a seventh aspect is a learning method for training a trained model to be installed in a computer, and includes the steps of: setting a first required value for a predetermined parameter of a communication device controlled by the computer using the trained model; training the trained model so that a reward to be awarded in a predetermined environment is maximized; extracting a trained model with a predetermined number of learning steps or more as a trained model; determining whether the performance of the extracted trained model has reached the first required performance; if it is determined that the performance of the trained model has reached the first required performance, updating the first required value to a second required value different from the first required value; determining whether the performance of the trained model updated to the second required value satisfies the second required performance different from the first required performance; and if it is determined that the performance of the trained model updated to the second required value satisfies the second required performance, selecting the trained model that satisfies the second required performance as the trained model to be installed in the computer. As a result, the learning method of the seventh aspect can appropriately select a trained model that satisfies the required performance to make an unmanned aircraft system viable, and can further improve performance that is difficult to achieve with conventional wireless communication design alone.

[0063] The learning program of the eighth aspect is a learning program for using a learning device to learn a trained model installed in a computer, and causes the learning device to execute the following steps: setting a first required value for a predetermined parameter of a communication device controlled by the computer using the trained model; training the trained model so as to maximize the reward awarded in a predetermined environment; extracting a trained model with a predetermined number of learning steps or more as a trained model; determining whether the performance of the extracted trained model has reached the first required performance; if it is determined that the performance of the trained model has reached the first required performance, updating the first required value to a second required value different from the first required value; determining whether the performance of the trained model updated to the second required value satisfies the second required performance different from the first required performance; and, if it is determined that the performance of the trained model updated to the second required value satisfies the second required performance, selecting the trained model that satisfies the second required performance as the trained model to be installed in the computer. As a result, the learning device of the eighth aspect can appropriately select a trained model that satisfies the required performance to make an unmanned aircraft system viable, and can further improve performance that is difficult to achieve with conventional wireless communication design alone.

[0064] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0065] 10 Learning Device 12 Environment Department 14 Storage section 16 Control Unit 20 Exercise Model 22 Environmental Model 24 Remuneration Model 30 Reinforcement Learning Models 40 Setting section 42 Learning Department 50 Reinforcement Learning Department 52 Model Extraction Unit 54 Model Evaluation Unit 56 Update section 58 Model Selection Section

Claims

1. A learning device for learning a trained model installed in a computer, a setting unit that sets a first required value for a predetermined parameter of a communication device controlled by the computer using the trained model; a reinforcement learning unit that trains a learning model so as to maximize the reward given in a predetermined environment; a model extraction unit that extracts the learning model having a predetermined number of learning steps or more as a trained model; a model evaluation unit that determines whether the performance of the trained model extracted by the model extraction unit reaches a first required performance; an update unit that updates the first required value to a second required value different from the first required value when the model evaluation unit determines that the performance of the trained model has reached the first required performance; a model selection unit that selects the trained model to be installed in the computer; Equipped with the model evaluation unit determines whether the performance of the trained model updated to the second required value satisfies a second required performance that is different from the first required performance; the model selection unit, when the model evaluation unit determines that the performance of the trained model updated to the second required value satisfies the second required performance, selects the trained model that satisfies the second required performance as the trained model to be installed in the computer; the predetermined parameter of the communication device is an information transfer rate; the setting unit sets a first transmission rate as the first required value for the information transfer rate; the updating unit updates the first transmission rate to a second transmission rate that is slower than the first transmission rate. Learning device.

2. the update unit changes the second required value in accordance with the second required performance. The learning device according to claim 1 .

3. A communication device that performs communication based on control using a trained model trained by the learning device according to claim 1 or 2.

4. a computer equipped with a trained model trained by the learning device according to claim 1 or 2; a communication device, The computer uses the trained model to communicate via the communication device. Drone.

5. A wireless communication system including a plurality of unmanned aerial vehicles according to claim 4.

6. A learning method for learning a trained model installed in a computer using a learning device, comprising: setting a first requirement value for a predetermined parameter of a communication device controlled by the computer using the trained model; A step of training a learning model so as to maximize the reward given in a given environment; extracting the learning model having a predetermined number of learning steps or more as a trained model; determining whether the performance of the extracted trained model has reached a first required performance; When it is determined that the performance of the trained model has reached the first required performance, updating the first required value to a second required value different from the first required value; determining whether the performance of the trained model updated to the second required value satisfies a second required performance that is different from the first required performance; When it is determined that the performance of the trained model updated to the second required value satisfies the second required performance, selecting the trained model that satisfies the second required performance as the trained model to be installed in the computer; Including, the predetermined parameter of the communication device is an information transfer rate; In the setting step, a first transmission rate is set as the first required value for the information transfer rate; In the updating step, the first transmission rate is updated to a second transmission rate that is slower than the first transmission rate. How to learn.

7. A learning program for learning a trained model installed in a computer using a learning device, setting a first requirement value for a predetermined parameter of a communication device controlled by the computer using the trained model; A step of training a learning model so as to maximize the reward given in a given environment; extracting the learning model having a predetermined number of learning steps or more as a trained model; determining whether the performance of the extracted trained model has reached a first required performance; When it is determined that the performance of the trained model has reached the first required performance, updating the first required value to a second required value different from the first required value; determining whether the performance of the trained model updated to the second required value satisfies a second required performance that is different from the first required performance; When it is determined that the performance of the trained model updated to the second required value satisfies the second required performance, selecting the trained model that satisfies the second required performance as the trained model to be installed in the computer; causing the learning device to execute the predetermined parameter of the communication device is an information transfer rate; In the setting step, a first transmission rate is set as the first required value for the information transfer rate; In the updating step, the first transmission rate is updated to a second transmission rate that is slower than the first transmission rate. Learning program.

Images