Learning device, learning method, and program

Abstract

In this learning device, an instruction means instructs a route simulator with an action relating to the flight of an flying body. An observation value acquisition means acquires, from the route simulator, an observation value of the state of a route resulting from the action. A reward calculation means calculates a reward on the basis of the observation value. A learning means learns a policy function that outputs an action on the basis of the input of an observation value so as to maximize the reward. An action selection means selects a next action with respect to the state of the route by using the policy function.

Description

Learning device, learning method, and recording medium

[0001] The present disclosure relates to air traffic control using AI (Artificial Intelligence).

[0002] Patent Document 1 describes a method for generating a statistical model that outputs aviation instruction data using previously issued air traffic control instruction data and past flight track data and aviation environment data corresponding to the air traffic control instruction data as learning data.

[0003] Japanese Patent Application Laid-Open No. 2006-202046

[0004] One objective of the present disclosure is to provide a system that can support air traffic control operations by using AI to automatically and quickly develop optimal approach avoidance plans and present them to air traffic controllers.

[0005] In one aspect of the present disclosure, a learning device comprises: an instruction means for instructing a route simulator on an action related to the flight of an aircraft; an observation value acquisition means for acquiring from the route simulator observed values ​​of the route state resulting from the action; a reward calculation means for calculating a reward based on the observed values; a learning means for learning a policy function that outputs an action based on input of the observed values ​​so as to maximize the reward; and an action selection means for selecting a next action for the route state using the policy function.

[0006] In another aspect of the present disclosure, a computer-implemented learning method includes instructing a flight route simulator to take actions related to the flight of an aircraft; obtaining observations of flight route states resulting from the actions from the flight route simulator; calculating a reward based on the observations; learning a policy function that outputs an action based on the input of the observations so as to maximize the reward; and using the policy function to select a next action for the flight route state.

[0007] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute the following processes: instructing a route simulator to take actions related to the flight of an aircraft; obtaining from the route simulator observed values ​​of route conditions resulting from the actions; calculating a reward based on the observed values; learning a policy function that outputs an action based on the input of the observed values ​​so as to maximize the reward; and using the policy function to select the next action for the route conditions.

[0008] 1 is a schematic diagram showing an example of an aircraft trajectory; FIG. 2 is a diagram showing the overall configuration of a learning system according to a first embodiment; FIG. 3 is a block diagram showing the hardware configuration of a learning device according to a first embodiment; FIG. 4 is a block diagram showing the functional configuration of a learning device according to a first embodiment; FIG. 5 is a diagram explaining observed values ​​of route conditions obtained by a route simulator; FIG. 6 is a diagram showing an example of a reward function; FIG. 7 is a flowchart of learning processing; FIG. 8 is a diagram showing an example of the configuration of a control system using an AI control model during actual operation; FIG. 9 is a block diagram showing the configuration of a learning device according to a second embodiment; FIG. 10 is a flowchart of processing by a learning device according to a second embodiment.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present disclosure will now be described with reference to the accompanying drawings. DESCRIPTION OF THE PRESENT INVENTION In order to facilitate understanding of the present disclosure, the prior art and problems that are the premise of the present disclosure will be described.

[0010] Japan's air traffic control operations are classified into airport control, terminal control, enroute control, and ocean control. Pilots follow airport control until the aircraft takes off from the airport, enroute control while ascending to cruising altitude, and terminal control while descending near the airport. Pilots also follow ocean control when entering an ocean control area over the Pacific Ocean. To prevent aircraft from getting too close to each other, controllers instruct each aircraft to change their altitude, course, etc. to ensure that they always maintain a certain distance apart (control separation).

[0011] In recent years, the number of aircraft is expected to increase due to factors such as growing demand for international air transport. If the number of aircraft flying in the airspace is n, then the number of pairs of two aircraft is nC2 = n(n-1) / 2. Therefore, as the number of aircraft increases, the number of candidates for approach and collision increases exponentially. As the number of aircraft increases, the difficulty of formulating an approach avoidance plan for aircraft increases, so a function to automatically support the formulation of approach avoidance plans is required to formulate optimal approaches avoidance plans.

[0012] One objective of the present disclosure is to provide a system that can support air traffic control operations by using AI to automatically and quickly develop optimal approach avoidance plans and present them to air traffic controllers.

[0013] In the following, this disclosure will be described using an aircraft as an example of a target of air traffic control operations, but the air traffic control operations supported by the system are not limited to aircraft. The target of air traffic control operations may be any mobile object moving in a three-dimensional space, as described below. For example, the target of air traffic control operations supported by the system may be a mobile object moving underwater, such as a submarine, or an air vehicle, such as a small unmanned aircraft like a drone.

[0014] <Explanation of Related Matters> First, matters related to the present disclosure will be described. [Trajectory] Trajectory is a fundamental concept in modern air traffic control. A "trajectory" is the movement path of an aircraft, and is a set of tuples (time series data) consisting of latitude, longitude, altitude, and time, or these in addition to speed, acceleration, etc. Because aircraft can fly freely in three-dimensional space, the movement path of an aircraft, including time, is a line in four-dimensional space, and this line is called a trajectory. Figure 1 schematically shows an example of an aircraft trajectory.

[0015] [Dynamic System] A system whose situation changes over time, such as an airspace where the positions of many aircraft change from moment to moment, is called a dynamical system. In this disclosure, among dynamical systems, a discrete-time dynamical system is used. A discrete-time dynamical system is expressed by a state transition function f: X × U → X, where X is a state set and U is an input set, and for time t ∈ Z, the state changes as x(t + 1) = f(x(t), u(t)).

[0016] [Reinforcement Learning] Reinforcement learning is a method for solving optimal control problems. Given a dynamical system to be controlled and a "reward function" defined to return a larger value the more desirable the state at each time, reinforcement learning automatically constructs an operation method (policy = controller) that maximizes the sum of the reward function values ​​over the entire operation period (cumulative reward value). Reinforcement learning involves (1) using a learning controller to operate the dynamical system to collect input / output data for the system, and (2) using the collected data and reward function values ​​to adjust the controller to more easily select the operation that maximizes the cumulative reward value. Note that a controller that is not yet fully trained may cause undesirable conditions in the system. Furthermore, it is necessary to collect a large amount of data required for learning in a short period of time. For this reason, multiple parallel dynamic simulators capable of performing predictive calculations faster than reality are often used for learning rather than the actual system.

[0017] <First embodiment> [Overall configuration] Fig. 2 shows the overall configuration of a learning system according to the first embodiment. The learning system 1 comprises a learning device 100 and a sea route simulator 200. Although Fig. 1 shows one sea route simulator 200, the sea route simulator 200 may be configured with multiple simulators connected in parallel in order to efficiently perform learning using a large amount of data as described above.

[0018] The flight route simulator 200 is a dynamic simulator for reproducing airspace conditions and predicts future airspace conditions based on the current trajectory. In this embodiment, in order to perform reinforcement learning, the flight route simulator 200 outputs the state at equal time intervals (also called "time steps"), then pauses the simulation, and resumes the simulation when an instruction corresponding to the next time is input. The flight route simulator 200 also has a function for detecting the approach of each aircraft, and assigns an approach detection flag to each aircraft.

[0019] First, the initial state of the route is input to the route simulator 200. The initial state is, for example, the initial position of each aircraft on the route. Based on the initial state, the route simulator 200 predicts the position and speed (trajectory) of each aircraft at the next time, and outputs the predicted state to the learning device 100 as a state to be used in reinforcement learning.

[0020] The learning device 100 uses reinforcement learning to learn a model for performing AI control (hereinafter also referred to as the "AI control model"). The learning device 100 uses the AI ​​control model to determine an action that is considered optimal in an input state, and outputs it to the route simulator 200 as a control operation. The route simulator 200 predicts the state at the next time based on the control operation input from the learning device 100, and outputs it to the learning device 100. By repeating this process, the learning device 100 generates an AI control model that can determine the optimal control operation. Details of learning by the learning device 100 will be described later.

[0021] 3 is a block diagram showing the hardware configuration of the learning device 100. As shown in the figure, the learning device 100 includes an interface (IF) 12, a processor 13, a memory 14, a recording medium 15, and a database (DB) 16.

[0022] The IF 12 acquires observed values ​​indicating the state of the route from the route simulator 200. The IF 12 also outputs to the route simulator 200 the control operations generated by the AI ​​control model.

[0023] The processor 13 is a computer such as a CPU (Central Processing Unit) and executes a pre-prepared program to control the entire learning device 100. The processor 13 may be a CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof. The processor 13 executes the learning process described below.

[0024] The memory 14 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 14 stores various programs executed by the processor 13. The memory 14 is also used as a working memory while the processor 13 is executing various processes.

[0025] Recording medium 15 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from learning device 100. Recording medium 15 records various programs executed by processor 13. When learning device 100 executes various processes, the programs recorded on recording medium 15 are loaded into memory 14 and executed by processor 13.

[0026] The DB 16 stores the state of the route at each time input from the route simulator 200. The DB 16 also stores the control operation output by the learning device 100 to the route simulator 200 at each time.

[0027] In addition to the above, study device 100 may also include a display device such as a liquid crystal display or projector, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by an administrator of study device 100 to enter necessary operational inputs.

[0028] 4 is a block diagram showing the functional configuration of the learning device 100. Functionally, the learning device 100 includes a learning unit 20 and a simulation instruction unit 26. The learning unit 20 includes an observation value acquisition unit 21, a convolution unit 22, a reward calculation unit 23, an action selection unit 24, and a mask unit 25. Note that the learning device 100 is actually configured using a neural network.

[0029] The observation value acquisition unit 21 receives the observation values ​​of the route state output from the route simulator 200. Figure 5 is a diagram illustrating the route state representation by the route simulator 200, i.e., the observation values ​​of the route state. Although the number of aircraft flying along the route varies, it is preferable for the size of the observation values ​​to be fixed in reinforcement learning. Therefore, in this embodiment, the airspace is first divided into equally spaced rectangular regions (referred to as "boxes") in each direction of latitude, longitude, and altitude. Here, each box is sized so that no more than two aircraft can enter, in accordance with the definition of the control separation. In other words, only one aircraft can be present in each box B. Each aircraft is then assigned to a box, and the size of the observation values ​​is set to a fixed length by setting the observation value size = (total number of boxes) × (number of attribute values ​​of the box).

[0030] The observation values ​​were fourth-order tensors of latitude, longitude, altitude, and time, and each box contained the following three attribute values: (a) Latitude speed of the aircraft present in that box (b) Longitude speed of the aircraft present in that box (c) Approach detection flag Note that the speed is represented by the difference in coordinates between successive time steps. Furthermore, for boxes where no aircraft are present, the value of each attribute value is set to "0."

[0031] The convolution unit 22 performs a three-dimensional convolution operation on the observation values ​​acquired by the observation value acquisition unit 21. For example, the convolution unit 22 performs a convolution operation on multiple boxes set on the route using a three-dimensional filter of a predetermined size. As described below, the control operations instructed by the learning device 100 to the route simulator 200 basically relate to boxes in which aircraft are present. Therefore, if learning is performed using the observation values ​​of individual boxes as they are, learning may be insufficient for boxes in which no aircraft entered during learning, and the AI ​​control model may not be able to output appropriate approach avoidance plans for the airspace corresponding to those boxes. Therefore, the convolution unit 22 performs a three-dimensional convolution operation on the data of each box included in the observation values ​​to extract features, which are then used in the action selection unit 24 to learn a strategy. As a result, the AI ​​control model obtained by learning can generate appropriate approach avoidance plans based on the relative positional relationships of each aircraft, without relying on the absolute coordinates of the boxes in which aircraft are present.

[0032] The reward calculation unit 23 uses a prepared reward function to calculate a reward based on the observation value after the convolution operation. FIG. 6 shows an example of the reward function. In FIG. 6, the vertical axis represents the reward, and the horizontal axis represents the closest approach distance. The "closest approach distance" refers to the distance between the closest pair of aircraft among multiple aircraft at the same altitude. The reward function in FIG. 6 outputs a reward of "0" as no problem if the closest approach distance is 50 nautical miles or more. Furthermore, if the closest approach distance is less than 50 nautical miles, this reward function outputs a larger negative value as the reward, the smaller the closest approach distance.

[0033] The masking unit 25 has a function of limiting the boxes that are the targets of the action (control operation) selected by the action selecting unit 24. Reducing the number of boxes that are the targets of the action can improve the efficiency of reinforcement learning. Specifically, the masking unit 25 masks boxes in the observation values ​​acquired by the observation value acquiring unit 21 other than the box in which the approaching aircraft is detected (hereinafter referred to as the "approach detection box"). That is, the masking unit 25 references the attribute values ​​of each box included in the observation values ​​and masks boxes other than the approach detection box whose approach flag indicates approach. As a result, only unmasked (non-masked) boxes are the targets of the action selected by the action selecting unit 24. Note that the masking unit 25 may not mask boxes within a predetermined range near the approach detection box in addition to the approach detection box, and may mask boxes other than those boxes. Furthermore, the masking unit 25 may adjust the range of boxes to be masked depending on the task.

[0034] The action selection unit 24 learns the AI ​​control model based on the observation values ​​after convolution and the reward calculated by the reward calculation unit 23. Specifically, the action selection unit 24 updates the policy function based on the observation values ​​and the reward so that an action (control operation) that maximizes the cumulative reward value is more likely to be selected. Then, using the updated policy function, the action selection unit 24 determines the box that will be the target of the action in the next time step and the action (control operation) for that box, and outputs the results to the simulation instruction unit 26.

[0035] Specifically, the action selection unit 24 determines a box to be the target of action at each time step. Here, first, the action selection unit 24 selects only boxes in which an aircraft is present as the target of action. That is, the action selection unit 24 refers to the observation values ​​acquired by the observation value acquisition unit 21, and excludes boxes in which the above-mentioned three attributes (a) to (c) are all "0" from the target of action. Furthermore, the action selection unit 24 selects one or more boxes that are not masked by the masking unit 25 from among the multiple boxes in which aircraft are present as the target of action. In this way, by selecting only boxes in which an aircraft is present and in which the aircraft is approaching as the target of action, the efficiency of reinforcement learning can be improved.

[0036] The action selection unit 24 also determines an action for each box that is the target of the action. Air traffic control has three methods for approach avoidance: altitude change, course change, and speed change. In this embodiment, the action (control operation) determined by the policy function is assumed to be an altitude change of the aircraft. Therefore, the action selection unit 24 selects one of different predetermined altitude change amounts for each box that is the target of the action, and outputs the selected action. The altitude change amount may be, for example, -20, -10, 0, 10, or 20 FL (Flight Level). A positive value for the altitude change amount indicates an aircraft ascent, and a negative value indicates an aircraft descent. Note that 1 FL = 304.8 m. In this case, the policy function is a function that outputs the probabilities of five actions corresponding to the above five altitude change amounts.

[0037] The simulation instructing unit 26 outputs a simulation instruction to the flight route simulator 200 to cause the flight route simulator 200 to perform the control operation output as an action by the action selecting unit 24. The flight route simulator 200 performs a simulation of the control operation selected by the action selecting unit 24 in accordance with the input simulation instruction. For example, assume that the action selecting unit 24 instructs the control operation of "climbing 10 FL for box A where aircraft A is located" and the control operation of "descending 20 FL for box B where aircraft B is located." In this case, the flight route simulator 200 performs a simulation of ascending aircraft A by 10 FL and descending aircraft B by 20 FL in accordance with the simulation instruction. Then, the flight route simulator 200 generates a state at the next time step and outputs it to the learning device 100.

[0038] [Learning Process] Next, a description will be given of the learning process performed by the learning device 100. Fig. 7 is a flowchart of the learning process. This process is realized by the processor 13 shown in Fig. 3 executing a program prepared in advance and operating as each element shown in Fig. 4.

[0039] First, the route simulator 200 simulates a state corresponding to the initial state of the route, and the learning unit 20 acquires observed values ​​of the route state (step S11). Next, the learning unit 20 performs a three-dimensional convolution operation on the acquired observed values ​​(step S12). Next, the learning unit 20 uses a reward function prepared in advance to calculate a reward based on the observed values ​​after the convolution operation (step S13). Furthermore, the learning unit 20 extracts only boxes in which aircraft are present based on the observed values ​​acquired in step S11, and further masks boxes other than the box in which an approaching aircraft is present (step S14). Note that step S14 may be executed before steps S12 to S13, or may be executed in parallel with steps S12 to S13.

[0040] Next, the learning unit 20 updates the policy function based on the convolved observation value and the reward (step S15), and selects an action corresponding to the next time step using the updated policy function (step S16). At this time, the learning unit 20 targets one or more of the boxes not masked in step S14, and determines an action for each target box. The simulation instruction unit 26 then outputs a simulation instruction to the flight route simulator 200 to cause the aircraft belonging to the target box to execute the determined action. In this manner, steps S11 to S17 are repeated, and the learning process ends when a predetermined termination condition is met.

[0041] [Configuration during actual operation] Fig. 8 shows an example of the configuration during actual operation of a control system using a control model obtained by learning. The control system 1x includes a control assistance device 100x and a route simulator 200. The control assistance device 100x outputs control operations using a trained AI control model trained using the above-mentioned learning device 100. The route simulator 200 is the same as that of the learning system 1.

[0042] During actual operation, the current state is input from the actual control system to the route simulator 200. In addition, the control operation determined by the traffic control assistance device 100x is input to the route simulator 200. The route simulator 200 predicts the state at the next time step based on the input current state and control operation, and presents it to the controller and inputs it to the traffic control assistance device 100x. The traffic control assistance device 100x creates an optimal control operation (approach avoidance plan) based on the input state, and presents it to the controller and inputs it to the route simulator 200. In this way, the traffic control assistance device 100x can present the current state and the optimal control operation for the current state to the controller.

[0043] [Variations] (Variation 1) In the above embodiment, the control operation (action) output by the AI ​​control model (policy function) is an altitude change, but instead of or in addition to that, a speed change or an offset change may be output as the control operation.

[0044] (Variation 2) Furthermore, learning device 100 may improve the efficiency of learning by eliminating from the options actions that are ineffective depending on individual situations. For example, if there are basically 10 options for actions based on a policy function, and it is known that three of the 10 options are ineffective as actions in the current situation, learning device 100 can eliminate those three options in that situation and select an action from the remaining seven options.

[0045] (Variation 3) In actual operation of a route, a specific airspace may be temporarily designated as a no-fly zone. In such a case, reinforcement learning by the learning device 100 may also be performed to prevent aircraft from entering the restricted airspace. As a specific method, for example, a reward function may be set so that the reward for entering the restricted airspace is a large negative value.

[0046] (Variation 4) In the above embodiment, the method of the present disclosure is applied to air traffic control to prevent aircraft from approaching each other. However, the method of the present disclosure can also be applied to optimizing the flight plan of each aircraft before flight. Specifically, a simulation is performed based on the flight plan submitted in advance by each airline, and the submitted flight plan is changed to an optimized flight plan using reinforcement learning. This makes it possible to bring the entire airspace closer to the ideal situation.

[0047] <Application Example> Hereinafter, an application example in which this embodiment is applied to the field of medicine and healthcare will be described.

[0048] This embodiment can be applied to a medical supply delivery system using drones (hereinafter simply referred to as a "delivery system"). Specifically, the delivery system operates as follows. (1) First, the delivery system accepts an order for medical supplies. Specifically, a medical institution or medical personnel requests delivery of medical supplies online or by telephone. (2) Next, the delivery system manages and records delivery destination information. Specifically, the delivery system manages and updates delivery destination information, including the address and location information of the delivery destination, the desired delivery date and time, etc. The delivery destination information may include the medical condition and urgency of the patient to whom the medical supplies are to be delivered. Furthermore, the delivery system may take information about the medical condition and urgency into account when calculating the delivery route for the medical supplies. (3) Next, a pilot controls the flight of the drone based on the delivery route. At this time, the pilot acquires and updates the trajectory to be followed by the drone. The pilot may perform flight control taking into account the patient's medical condition and urgency. For example, the pilot may exclude the "slow down" option in flight control for drones carrying highly urgent supplies. When performing flight control, the drone pilot receives instructions from the control support device of this embodiment, thereby achieving the effects described in this embodiment. (4) Next, the delivery system manages the loading and descent of the delivery supplies. The delivery system then confirms that the medical supplies have been delivered by the drone, manages this, and updates the delivery destination information.

[0049] According to the delivery system, drones can be remotely controlled from an operational base such as a hospital to deliver medical supplies. This allows medical supplies to be delivered to depopulated areas without road or airport infrastructure. This delivery system does not require autonomous control or self-driving capabilities for the drones; instead, skilled pilots can remotely control the drones, ensuring safe delivery. Furthermore, by receiving flight control support from a control support device using the technology of this embodiment, optimized operation becomes possible even when multiple drones are in flight.

[0050] 9 is a block diagram showing the configuration of a learning device according to Embodiment 2. The learning device 70 includes an instruction unit 71, an observation value acquisition unit 72, a reward calculation unit 73, a learning unit 74, and an action selection unit 75.

[0051] 10 is a flowchart of processing by the learning device according to the second embodiment. The instruction means 71 instructs the route simulator on an action related to the flight of an aircraft (step S71). The observation value acquisition means 72 acquires, from the route simulator, observation values ​​of the route state resulting from the action (step S72). The reward calculation means 73 calculates a reward based on the observation values ​​(step S73). The learning means 74 learns a policy function that outputs an action based on the input of the observation values ​​so as to maximize the reward (step S74). The action selection means 75 uses the policy function to select the next action for the route state (step S76).

[0052] According to the learning device 70 of the second embodiment, it is possible to support air traffic control operations by proposing optimal actions based on the state of the route.

[0053] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0054] (Supplementary Note 1) A learning device comprising: an instruction means for instructing a route simulator on an action related to the flight of an aircraft; an observation value acquisition means for acquiring from the route simulator observed values ​​of the route state resulting from the action; a reward calculation means for calculating a reward based on the observed values; a learning means for learning a policy function that outputs an action based on input of the observed values ​​so as to maximize the reward; and an action selection means for selecting a next action for the route state using the policy function.

[0055] (Appendix 2) The state of the route is indicated by observation values ​​of a plurality of boxes of a predetermined size obtained by dividing the route at equal intervals in the latitude, longitude and altitude directions, and the predetermined size is a size that does not allow more than two aircraft to enter.

[0056] (Appendix 3) A learning device as described in Appendix 2, wherein the observation values ​​of the multiple boxes include the latitudinal speed of the aircraft present in each box, the longitudinal speed of the aircraft, and a proximity flag indicating whether the aircraft is in proximity to another aircraft.

[0057] (Supplementary Note 4) The learning device according to Supplementary Note 2, wherein the action selection means detects a box in which an aircraft is present based on the observation values ​​of the plurality of boxes, and determines the next action for one or more boxes in which the aircraft is present.

[0058] (Appendix 5) The learning device described in Appendix 4, wherein the plurality of box observation values ​​include the latitudinal speed of the aircraft present in each box, the longitudinal speed of the aircraft, and a proximity flag indicating whether the aircraft is in close proximity to another aircraft, and a box in which no aircraft is present is a box in which the latitudinal speed and the longitudinal speed are 0 and the proximity flag indicates that the aircraft is not in close proximity.

[0059] (Supplementary Note 6) The learning device according to Supplementary Note 2 further comprises a masking means for masking boxes other than the box to which the flying object is approaching based on the observation values ​​of the plurality of boxes, and the action selection means determines the next action for the unmasked boxes.

[0060] (Supplementary Note 7) The learning device according to Supplementary Note 2, further comprising: a calculation means for performing a three-dimensional convolution operation on the observed values ​​of the plurality of boxes; and the learning means for performing learning using the observed values ​​after the convolution operation.

[0061] (Appendix 8) The learning device according to Appendix 1, wherein the reward calculation means calculates the reward using a reward function that outputs a larger value the shorter the distance between a pair of aircraft that are closest to each other among a plurality of aircraft at the same altitude, when the distance between the pair of aircraft is shorter than a predetermined distance.

[0062] (Supplementary Note 9) The learning device according to claim 1, wherein the aerial vehicle includes a drone that delivers medical supplies to a patient, and the action selection means selects a next action taking into consideration the patient's condition or urgency.

[0063] (Supplementary Note 10) A learning method executed by a computer, comprising: instructing a route simulator to take actions related to the flight of an aircraft; obtaining from the route simulator observed values ​​of route states resulting from the actions; calculating a reward based on the observed values; learning a policy function that outputs an action based on the input of the observed values ​​so as to maximize the reward; and using the policy function to select a next action for the route state.

[0064] (Supplementary Note 11) A recording medium having recorded thereon a program that causes a computer to execute the following process: instructing a route simulator to take actions related to the flight of an aircraft; obtaining from the route simulator observed values ​​of route conditions resulting from the actions; calculating a reward based on the observed values; learning a policy function that outputs an action based on the input of the observed values ​​so as to maximize the reward; and using the policy function to select the next action for the route conditions.

[0065] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0066] REFERENCE SIGNS LIST 1 Learning system 13 Processor 21 Observation value acquisition unit 22 Convolution unit 23 Reward calculation unit 24 Action selection unit 25 Mask unit 26 Simulation instruction unit 100 Learning device 100x Air traffic control support device 200 Route simulator

Claims

1. an instruction means for instructing the flight route simulator on actions related to the flight of the aircraft; an observation value acquisition means for acquiring, from the route simulator, an observation value of the state of the route resulting from the action; a reward calculation means for calculating a reward based on the observed value; learning means for learning a policy function that outputs an action based on the input of the observation value so as to maximize the reward; an action selection means for selecting a next action for the state of the route using the policy function; A learning device comprising:

2. the state of the route is indicated by observation values ​​of a plurality of boxes of a predetermined size obtained by dividing the route at equal intervals in the latitude, longitude and altitude directions; The learning device according to claim 1 , wherein the predetermined size is a size that does not allow two or more flying objects to enter.

3. The learning device of claim 2, wherein the observation values ​​of the multiple boxes include the latitudinal speed of the aircraft present in each box, the longitudinal speed of the aircraft, and an approach flag indicating whether the aircraft is in approach to another aircraft.

4. The learning device according to claim 2 , wherein the action selection means detects a box in which an aircraft is present based on the observed values ​​of the plurality of boxes, and determines the next action for one or more boxes in which the aircraft is present.

5. the observation values ​​of the plurality of boxes include a latitude speed of the aircraft present in each box, a longitude speed of the aircraft, and a proximity flag indicating whether the aircraft is in a state of proximity to another aircraft; 5. The learning device according to claim 4, wherein the box in which the flying object does not exist is a box in which the latitude speed and the longitude speed are 0 and the approach flag indicates that the flying object is not in an approach state.

6. a masking means for masking boxes other than the box in which the flying object is approaching based on the observation values ​​of the plurality of boxes; 3. The learning device according to claim 2, wherein the action selection means determines the next action for an unmasked box.

7. a calculation means for performing a three-dimensional convolution calculation on the observed values ​​of the plurality of boxes; The learning means performs learning using the observation values ​​after the convolution operation. The learning device according to claim 2 .

8. The learning device according to claim 1, wherein the reward calculation means calculates the reward using a reward function that outputs a larger value the shorter the distance between a pair of aircraft that are closest to each other among a plurality of aircraft present at the same altitude, when the distance between the pair of aircraft is shorter than a predetermined distance.

9. the aerial vehicle includes a drone that delivers medical supplies to a patient; 2. The learning device according to claim 1, wherein the action selection means selects the next action in consideration of the patient's condition or urgency.

10. 1. A computer-implemented method of learning, comprising: Instruct the flight simulator on actions related to the flight of the aircraft, obtaining from the route simulator observations of route conditions resulting from the actions; calculating a reward based on the observations; A policy function that outputs an action based on the input of the observation value is learned so as to maximize the reward; A learning method that uses the policy function to select a next action for the state of the route.

11. Instruct the flight simulator on actions related to the flight of the aircraft, obtaining from the route simulator observations of route conditions resulting from the actions; calculating a reward based on the observations; A policy function that outputs an action based on the input of the observation value is learned so as to maximize the reward; A program that causes a computer to execute a process of selecting a next action for the state of the route using the policy function.