Information processing device, information processing method, program, and system

Abstract

To provide a technique for reducing a work load by a user monitoring a monitoring target object.SOLUTION: An information processing device includes a monitoring target object detection part for detecting a position of a monitoring target object in a photographic image or a screen, a monitoring important area calculation part for calculating a position of a monitoring important area of a user in the photographic image or the screen, and a first incidental information arrangement part for determining a display position of first incidental information being information related to the monitoring target object at positions corresponding to the position of the monitoring target object and the position of the monitoring important area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a program Mu and and systems. [Background technology]

[0002] Various technologies have been proposed to allow users to monitor objects. One example is a technology for air traffic controllers to monitor aircraft. For example, Patent Document 1 proposes a control support device that uses an electronic flight sheet, which is an electronic version of the paper flight sheet used in air traffic control operations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-134032 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is desirable to provide a technique for reducing the workload of a user who monitors a monitored object. Therefore, an object of the present invention is to provide a technique for reducing the workload of a user who monitors a monitored object. [Means for solving the problem]

[0005] In order to solve the above problem, according to one aspect of the present invention, a monitoring object detection unit detects the position of a monitoring object in a captured image or a screen, and a monitoring target detection unit detects the position of a user's monitoring focus area in the captured image or the screen. Based on the information of the user's gaze point generated by the gaze sensor a position calculation unit for calculating a position of the object to be monitored according to the position of the monitoring focus area; and a position adjacent to the position of the key monitoring area. and a first accessory information placement unit that determines a display position of first accessory information that is information related to the object to be monitored.

[0006] The information processing device may include an emphasis line calculation unit that calculates the position of an emphasis line, which is at least a part of a line passing through the position of the monitored object and the position of the monitoring focus area, and the first auxiliary information placement unit may determine the display position of the first auxiliary information based on the position of the emphasis line.

[0007] The information processing device detects the position of the monitored object. Nearby At the location, information relating to the monitored object is This is information different from the first auxiliary information. No. 2nd The display device may further include a second auxiliary information placement unit that determines the display position of the band information.

[0008] The second auxiliary information placement unit is configured to and at a position close to the position of the object to be monitored. A display position of the second incidental information may be determined.

[0010] The second incidental information placement unit may determine the display position of the second incidental information based on the fact that the user's point of gaze is within or near the image area of ​​the monitored object in the captured image or the screen.

[0012] The information processing device may include a transmission control unit that transmits data in which the first incidental information is arranged at a display position of the first incidental information determined by the first incidental information arrangement unit to a presentation control device via a communication unit.

[0013] According to another aspect of the present invention, there is provided a method for detecting a position of a monitored object in a captured image or a screen, and image The location of the user's monitoring focus area in Based on the information of the user's gaze point generated by the gaze sensor and calculating a position according to the position of the object to be monitored and the position of the key monitoring area. and a position adjacent to the position of the key monitoring area. and determining a display position of first auxiliary information, which is information related to the object to be monitored.

[0014] According to another aspect of the present invention, a computer includes a monitoring object detection unit that detects the position of a monitoring object in a captured image or a screen, and a monitoring target detection unit that detects the position of a user's monitoring focus area in the captured image or the screen. Based on the information of the user's gaze point generated by the gaze sensor a position calculation unit for calculating a position of the object to be monitored according to the position of the monitoring focus area; and a position adjacent to the position of the key monitoring area. and a first incidental information placement unit that determines a display position of first incidental information that is information related to the monitored object.

[0016] According to another aspect of the present invention, a monitoring target object detection unit detects the position of a monitoring target object in a captured image or a screen, and a monitoring target object detection unit detects the position of a user's monitoring focus area in the captured image or the screen. Based on the information of the user's gaze point generated by the gaze sensor a position calculation unit for calculating a position of the object to be monitored according to the position of the monitoring focus area; and a position adjacent to the position of the key monitoring area. and a first auxiliary information arrangement unit that determines a display position of first auxiliary information that is information related to the object to be monitored, Attachment 1 A system is provided that includes a presentation control device that includes a control unit that controls a display unit so that band information is displayed. [Effects of the Invention]

[0017] As described above, the present invention provides a technique for reducing the workload of a user who monitors a monitoring target object. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is an explanatory diagram showing an example of a schematic configuration of an air traffic control assistance system according to an embodiment of the present invention; [Figure 2] 2 is a block diagram showing an example of the hardware configuration of an air traffic control support server, an airport camera, an air traffic control information presentation device, and an aviation information management system according to the embodiment. FIG. [Figure 3]FIG. 2 is a block diagram illustrating an example of a functional configuration of the traffic control information presentation device according to the embodiment. [Figure 4] 2 is an explanatory diagram for explaining an example of air traffic control support information displayed on the screen (display unit) of the air traffic control information presentation device. FIG. [Figure 5] 10 is an explanatory diagram for explaining an example of the display positions of the first auxiliary information and the second auxiliary information on the screen. FIG. [Figure 6] 2 is a block diagram showing an example of a functional configuration of a control assistance server 100 according to the embodiment. FIG. [Figure 7] FIG. 2 is an explanatory diagram showing an example of an operation flow of the air traffic control assistance system according to the embodiment. [Figure 8] FIG. 2 is an explanatory diagram showing an example of an operation flow of the air traffic control assistance system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0020] In this specification and drawings, multiple components having substantially the same or similar functional configurations are distinguished by adding different alphabets to the same reference numeral. However, when there is no need to particularly distinguish between multiple components having substantially the same or similar functional configurations, only the same reference numeral is used.

[0021] (0. Overview) Air traffic control is the process by which control agencies give instructions on flight methods and other matters to ensure the safe operation of aircraft (Kojien 6th edition), and it is a task that carries the risk of even a small human error leading to a serious aircraft accident.

[0022] Air traffic controllers (hereafter referred to as "controllers") responsible for air traffic control duties must simultaneously monitor the status of, in some cases, ten or more aircraft, and issue instructions and communications with the appropriate timing and content to each aircraft's pilots and other controllers. This work is known to place a high mental strain on the controller. As a result, controllers sometimes give incorrect instructions due to human error or fall asleep due to fatigue, resulting in aircraft accidents and serious incidents. The workload of such controllers and the resulting risk of human error and accidents and incidents must be addressed as a challenge.

[0023] With the development of information and communication technology, there are signs of two major changes in the environment in which air traffic controllers work.

[0024] The first change is the computerization of flight strips used by air traffic control, that is, the introduction of an electronic flight strip system. Patent Document 1 (JP 2016-134032 A) and other documents describe examples of electronic flight strips.

[0025] The second change is the realization of remote airport control systems using remote towers (hereafter referred to as remote airport control) (e.g., Electronic Navigation Research Institute, Practical Research on Remote Airport Operations Support Systems, https: / / www.enri.go.jp / research / kenkyu / M-17-01.html). In a remote airport control system, imaging devices such as wide-angle cameras are installed at airports, and video imagery of the airport captured by these cameras is presented to air traffic controllers located far from the airport via low-latency network communications. The controllers view video imagery including airport runways and aircraft displayed on large display devices, and communicate remotely with pilots of aircraft at remote airports to give instructions and communicate with them.

[0026] The aforementioned functions of remote airport control using electronic flight sheets and video communication are expected to be widely introduced into future airport control systems.

[0027] An embodiment of the present invention realizes a new remote airport control support system that can reduce the workload of controllers by applying the technical elements of the above-mentioned "electronic flight sheet" and "video communication at remote airports."

[0028] One element of a controller's workload is the high cognitive and memory processing load. Controllers must simultaneously grasp the status of multiple aircraft, look at the paper flight sheet on their desk or the electronic flight sheet displayed on a display device, memorize the information, and mentally link the status of the multiple aircraft with the information on the flight sheet (call signs, aircraft types, estimated departure / arrival times, etc.).

[0029] At the same time, controllers must visually monitor changes in the status of aircraft scattered at distances from each other within the airport, i.e., detect visual changes within the wide field of view. Through these and other tasks, controllers are constantly checking whether the status of the entire airport is changing according to plan.

[0030] Here, the former action of linking and memorizing the status of each aircraft with the information on the flight schedule can be reduced by using AR (Augmented Reality) technology to visually link and display a moving image of the aircraft with the electronic flight schedule information corresponding to the aircraft for each aircraft. In other words, by linking and displaying a moving image of the aircraft with the electronic flight schedule information corresponding to the aircraft on a display, the action of memorizing the link becomes unnecessary.

[0031] Furthermore, the latter action of detecting visual changes within the wide-field information of the airport can reduce the mental load on the controller by presenting AR information on the electronic flight schedule, which changes in response to visual changes within the wide-field information, in the vicinity of the controller's key monitoring area (described later). In other words, by presenting AR information on the electronic flight schedule in the vicinity of the controller's key monitoring area (described later), the controller's visual search range becomes smaller, allowing the controller to notice changes in the situation of the monitored object with a low visual load.

[0032] Here, we will also consider a situation in which multiple air traffic control support systems according to embodiments of the present invention are used by multiple air traffic controllers. Air traffic control operations are often performed as a collaborative effort by multiple air traffic controllers. For example, at airports with heavy air traffic, such as hub airports, there are cases in which air traffic controllers with multiple roles, such as airport controller, ground controller, control approval transmitter, assistant controller, and general controller, work together for each runway. In such a case of multiple air traffic control operations, for example, in a two-person system in which a first controller plays the role of airport controller or ground controller, and a second controller plays the role of assistant controller or general controller, responsible for liaison and coordination with other related organizations, we will consider the following case.

[0033] Because the first controller plays the role of airport controller or ground controller, it is useful to utilize the function of presenting AR information on the airport video near the first controller's key monitoring area (described below). On the other hand, the second controller plays the role of assistant controller or chief controller, and may intermittently view unspecified locations on the airport video in accordance with liaison and coordination with other related organizations. In other words, because the key monitoring area (described below) of the first controller is different from the key monitoring area of ​​the second controller, in this case, the location of the AR information presented near the first controller's key monitoring area (described below) is not particularly useful for the second controller, and it is suggested that presenting AR information near the video images of each aircraft would be more useful due to the location of the AR information presented.

[0034] In summary, the present invention proposes a technology for realizing a new remote airport control support system, which is characterized by detecting a video of a monitored object such as an aircraft on a video image of a remote airport, linking the detected video of the monitored object with related information such as an electronic flight ticket and presenting AR information, and further controlling the presentation position of the AR information so as to reduce the visual search target range of the first controller.In addition, the present invention proposes a technology for realizing a remote airport control support system, which is characterized by controlling the presentation of AR information having content equivalent to the AR information for the second controller so as to be also near the video of the monitored object.

[0035] In the embodiments of the present invention, an aircraft will be described as an example of a monitored object. The aircraft may be a manned aircraft or an unmanned aircraft (a so-called unmanned aircraft). However, the monitored object may also be an object other than an aircraft. For example, the monitored object may be a person, an animal other than a person, or an inanimate object (for example, a ship, a vehicle, a robot, etc.).

[0036] Furthermore, in the embodiment of the present invention, the occupation of the user monitoring the monitored object is described as an example of a traffic controller. However, the user monitoring the monitored object does not have to be a traffic controller. For example, the occupation of the user monitoring the monitored object may be changed as appropriate depending on the type of the monitored object.

[0037] (1. Description of the structure) First, referring to FIG. 1, a schematic configuration of an air traffic control support system according to an embodiment of the present invention will be described. FIG. 1 is an explanatory diagram showing an example of the schematic configuration of an air traffic control support system according to this embodiment. Referring to FIG. 1, the air traffic control support system includes an air traffic control support server 100, an airport camera 200, a line-of-sight sensor 300, an air traffic control information presentation device 400, an aviation information management system 500, and a LAN 99. FIG. 1 also illustrates an airport 10, a remote control room 30, an aircraft 60, and an air traffic controller 90 as elements of space, objects, and people for explaining the air traffic control support system. In this embodiment, the airport camera 200 and the aircraft 60 are located within or around the airport 10, and the line-of-sight sensor 300, the air traffic control information presentation device 400, and the air traffic controller 90 are located within the remote control room 30.

[0038] 2 is a block diagram showing an example of the hardware configuration of the air traffic control support server 100, airport camera 200, air traffic control information presentation device 400, and aviation information management system 500 according to this embodiment (hereinafter, these may be referred to as "devices according to this embodiment" without distinguishing between them). Note that not all of the above-mentioned devices need to be equipped with all of the hardware configuration described below (for example, the air traffic control support server 100 does not need to be directly equipped with sensors), and each device may be equipped with an appropriately limited number of hardware modules that can realize the functional configuration of each device described below.

[0039] 2, the device according to this embodiment includes a bus 801, a CPU (Central Processing Unit) 803, a ROM (Read Only Memory) 805, a RAM (Random Access Memory) 807, a storage device 809, a communication interface 811, a sensor 813, an input device 815, a display device 817, and a speaker 819. The CPU 803 executes various processes in the device according to this embodiment.

[0040] The ROM 805 also stores programs and data for causing the CPU 803 to execute processing in the device according to this embodiment. The RAM 807 also temporarily stores programs and data when the CPU 803 is executing processing. The bus 801 interconnects the CPU 803, the ROM 805, and the RAM 807. The bus 801 is further connected to a storage device 809, a communication interface 811, a sensor 813, an input device 815, a display device 817, and a speaker 819. The bus 801 includes, for example, multiple types of buses. As one example, the bus 801 includes a high-speed bus connecting the CPU 803, the ROM 805, and the RAM 807, and one or more other buses slower than the high-speed bus.

[0041] The storage device 809 stores data to be temporarily or permanently saved in the device according to this embodiment. The storage device 809 may be, for example, a magnetic storage device such as a hard disk, or may be a nonvolatile memory such as an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory, an MRAM (Magnetoresistive Random Access Memory), an FeRAM (Ferroelectric Random Access Memory), or a PRAM (Phase Change Random Access Memory).

[0042] The communication interface 811 is a communication means provided in the device according to this embodiment, and communicates with an external device via a network (or directly). The communication interface 811 may be an interface for wireless communication, and in this case, may include, for example, a communication antenna, an RF circuit, and other circuits for communication processing. The communication interface 811 may also be an interface for wired communication, and in this case, may include, for example, a LAN terminal, a transmission circuit, and other circuits for communication processing.

[0043] The sensor 813 is, for example, a camera, a microphone, a biosensor, or other sensor, or a combination thereof. The camera captures an image of a subject and includes, for example, an optical system, an image sensor, and an image processing circuit. The microphone collects ambient sound, converts the sound into an electrical signal, and then converts the electrical signal into digital data. The input device 815 is, for example, a touch panel, a mouse, or a gaze detection device. The display device 817 displays an output image (i.e., a display screen) from the device according to this embodiment and can be realized using, for example, a liquid crystal display, an organic light-emitting diode (OLED), a cathode ray tube (CRT), or the like. The speaker 819 outputs sound, converting digital data into an electrical signal and then converting the electrical signal into sound.

[0044] First, an example of the functional configuration of the "airfield camera 200" according to this embodiment will be described.

[0045] The airport camera 200 is a network camera that captures images of the airport 10 and aircraft 60 and outputs and transmits the generated video data to the air traffic control support server 100 via the LAN 99. To perform the above processing, the airport camera 200 may be functionally configured with a communication unit, a memory unit, a control unit, and the like in addition to an imaging unit (sensor unit), although these are not shown. The airport camera 200 may be a panoramic camera capable of capturing images with an ultra-wide angle of view of 180° or 360°, or a PTZ (Pan-Tilt-Zoom) camera with pan, tilt, and zoom functions. The airport camera 200 may be implemented by a sensor 813.

[0046] Next, an example of the functional configuration of the "gaze sensor 300" according to this embodiment will be described.

[0047] The line-of-sight sensor 300 generates "point of gaze" information indicating where on the screen (display unit) of the air traffic control information presentation device 400 (described later) the air traffic controller 90 in the remote control room 30 is looking, and transmits the information to the air traffic control support server 100 via the LAN 99. The information on the point of gaze includes, for example, position information indicated by two-dimensional (X, Y) coordinates on the screen (display unit) of the air traffic control information presentation device 400 and information on the time when the position information was generated.

[0048] Known gaze sensing methods use special hardware devices, such as the corneal reflex method using near-infrared light, the scleral reflex method, the electrooculography method, and the search coil method, as described in Non-Patent Document 1 (Non-Patent Document 1: Hashimura Masaru, Iizuka Hiromi, Li Jun, 2015, "Measurement of Eye Movement", Ergonomics, 51(6), pp.406-410).

[0049] Another known gaze sensing method is a method that combines a general camera device with an image processing method. As described in Non-Patent Document 2, this method includes an appearance-based method and a machine learning-based estimation method (Non-Patent Document 2: Yusuke Kanno, 2020, "Research Trends in Gaze Measurement and Estimation Technology," Journal of the Institute of Image Information and Television Engineers, 74(3), pp. 495-500).

[0050] In the embodiment of the present invention, any of the above-mentioned gaze sensing methods may be used as long as it can generate information on the gaze point of the air traffic controller 90A. Furthermore, there are two types of gaze sensors: an environmentally installed type and a head-mounted type. In this embodiment, the description will be mainly based on the environmentally installed type (for example, installed near the screen of the air traffic control information presentation device 400 described later). However, the type of gaze sensor may also be a head-mounted type. Furthermore, the environmentally installed type gaze sensor 300 may have information input in advance by an administrator of the system regarding the relative positional relationship in three-dimensional space between itself and the plane of the screen (display unit) of the air traffic control information presentation device 400.

[0051] Note that depending on the biological shape of the eyes and the operating environment of the gaze sensor, it may not be possible to measure the gaze with high accuracy. Alternatively, even if the gaze can be measured, there is a method for improving the estimation accuracy of gaze information by combining the gaze measured by the gaze sensor with information on the face direction to estimate the gaze. Therefore, the gaze sensor 300 may have a function for estimating the face direction as alternative information to the gaze of the controller 90. As an example of a face direction sensing method, as described in Non-Patent Document 3, a method for estimating the face direction using feature point information of a face image obtained from a general camera device is known (Non-Patent Document 3: Yamazaki Akito, Pongsathornraksilansak, and Otake Motoki, 2017, "Extracting the Gaze Area by Estimating the Driver's Face Direction Using an In-Vehicle Camera," Transactions of the Society of Automotive Engineers of Japan, 48(5), pp. 1113-1119).

[0052] From the estimated face direction information, for example, a three-dimensional vector extending in the face front direction, the gaze sensor 300 may calculate position information of the intersection point between the face front direction and the plane of the screen (display unit) of the air traffic control information presentation device 400, and may replace the position information of the intersection point with the above-mentioned information on the gaze point. Note that the gaze sensor 300 may be implemented by the sensor 813.

[0053] Next, an example of the functional configuration of the "aviation information management system 500" according to this embodiment will be described.

[0054] The aviation information management system 500 has a set of various information data necessary for the airfield control operations of the controllers 90, and may transmit a part of the data to the airfield control support server 100 in response to a request from the airfield control support server 100. For example, the set of various information data necessary for the airfield control operations may be acquired by the aviation information management system 500 from a system or database that handles the data.

[0055] The set of various information data required for airfield control operations may include data handled by the following systems and databases. That is, the systems may be FDP (Flight Data Processing System), RDP (Radar Data Processing System), MLAT (Multilateration: an in-airfield aircraft positioning device), etc. The databases may be aircraft information databases, sector information databases, etc.

[0056] Here, the FDP handles operational information of each aircraft present within the airport 10 or the surrounding airspace controlled by the controller 90 (for example, each aircraft's destination, planned flight distance, spot (parking) location, takeoff location, scheduled spot departure time, scheduled takeoff time, etc.). The RDP handles information such as the flight number and altitude of each aircraft. The MLAT handles the three-dimensional position and identification information (for example, call sign) of each aircraft 60. For example, the three-dimensional position of each aircraft 60 may be latitude, longitude, and altitude information, or three-dimensional relative position information between the airport camera 200 and each aircraft 60.

[0057] The aircraft information database handles data such as the name, air route, and flight performance information for each aircraft model of each aircraft 60. The sector information database handles data on air routes within the controlled airspace (sector), and data indicating the positions of fixes such as junctions, target points, and radio beacons.

[0058] In particular, the aviation information management system 500 may transmit to the air traffic control support server 100 the latitude, longitude, and altitude information of each aircraft 60 present within the airport 10 or the surrounding airspace that is under the control of the controller 90, or the three-dimensional relative position information between the airport camera 200 and each aircraft 60, in association with the identification information and operation information of each aircraft 60. The latitude, longitude, and altitude information and the three-dimensional relative position information of each aircraft 60 may be generated using the MLAT described above.

[0059] Next, an example of the functional configuration of the "traffic control information presentation device 400" according to this embodiment will be described with reference to Fig. 3. The traffic control information presentation device 400 includes a communication unit 410, a storage unit 420, a control unit 430, and a display unit 450. The traffic control information presentation device 400 can function as an example of a presentation control device.

[0060] The communication unit 410 communicates with other devices. For example, the communication unit 410 is directly connected to the LAN 99 and communicates with the air traffic control support server 100. More specifically, the communication unit 410 receives air traffic control support information data, which is generated by the air traffic control support server 100 (described later) and in which additional information is added to a captured video image of the airport 10 in a manner characteristic of the embodiment of the present invention. The communication unit 410 can be implemented by a communication interface 811.

[0061] The storage unit 420 stores programs and data for the operation of the traffic control information presentation device 400. The storage unit 420 can be implemented by the storage device 809.

[0062] The control unit 430 provides various functions of the traffic control information presentation device 400. The control unit 430 can be implemented by the CPU 803, the ROM 805, and the RAM 807.

[0063] The display unit 450 presents information to the controller 90. For example, the display unit 450 has a screen, and displays the air traffic control support information on the screen based on the data of the air traffic control support information received by the communication unit 410, and presents it to the controller 90. Note that the display unit 450 can be realized by, for example, the display device 823.

[0064] 4 is an explanatory diagram for explaining an example A40 of air traffic control support information displayed on the screen (display unit) of the air traffic control information presentation device 400. Note that the dashed lines and center of gravity shown in FIG. 4 are graphic elements for explaining the present invention, and may not be displayed on the screen (display unit) of the actual air traffic control information presentation device 400.

[0065] Referring to Figure 4, there are shown a screen A40 in which control support information relating to an embodiment of the present invention is displayed on a moving image of an airport, a moving image A401 of an airport 10, an image A441 of an aircraft 60, a monitored object detection area A442, a monitoring priority area A451, an example of a focus point A452, a center of gravity A461 of the monitored object detection area, a center of gravity A462 of the monitoring priority area, a priority line A463, first ancillary information A471, second ancillary information A481, and a priority line A483 (those shown in multiple figures are numbered A, B, C, etc.).

[0066] 4, first incidental information A471 is visually associated with an image A441 of each aircraft 60 and displayed in AR, and the first incidental information A471 is displayed in the vicinity of a key monitoring area A451 of the controller 90A. A detailed description of each of the illustrated elements will be given in the description of the air traffic control support server 100, which will be described later.

[0067] In the embodiment of the present invention, it is mainly assumed that the controller 90 views the captured video A401 of the airport 10 displayed on the screen A40. However, the captured video A401 of the airport 10 does not have to be displayed. That is, the controller 90 may be located inside the airport 10, which is real space (not remote), or in a control tower located nearby, and view the airport 10 directly. In this case, the display unit 450 is configured with a transparent display, and the controller 90 views the airport 10, which is real space, through the transparent display instead of the airport 10 displayed on the screen A40. Then, the supplementary information is displayed superimposed on the airport 10, which is real space, instead of being superimposed on the airport 10 displayed on the screen A40.

[0068] Fig. 5 is an explanatory diagram for explaining an example of the display positions of the first incidental information A471 and the second incidental information A481 on the screen A40. Referring to Fig. 5, similar to Fig. 4, the screen A40, the captured video A401 of the airport 10, the image A441 of the aircraft 60, the monitored object detection area A442, the monitoring priority area A451, an example of the point of interest A452, the center of gravity A461 of the monitored object detection area, the center of gravity A462 of the monitoring priority area, the first incidental information A471, and the second incidental information A481 are shown.

[0069] 5, the center of gravity A472 of the first auxiliary information area, the center of gravity A482 of the second auxiliary information area, and the center of gravity A474 of the first auxiliary information area are also shown.

[0070] Next, an example of the functional configuration of the "control support server 100" according to this embodiment will be described with reference to Fig. 6. The control support server 100 can function as an example of an information processing device.

[0071] The air traffic control support server 100 acquires and receives, via the LAN 99, captured video image data of the airport 10 from the airport camera 200, information on the gaze point of the air traffic controller 90A from the line-of-sight sensor 300, and three-dimensional position information, identification information, and flight information of each aircraft 60 from the aviation information management system 500. The air traffic control support server 100 then generates data of air traffic control support information (information in which supplementary information is added to the captured video image of the airport 10 in a manner that is a feature of the embodiment of the present invention) and transmits it to the air traffic control information presentation device 400.

[0072] 6 is a block diagram showing an example of the functional configuration of the control support server 100 according to this embodiment. Referring to FIG. 6, the control support server 100 includes a communication unit 110, a storage unit 120, and a control unit .

[0073] The communication unit 110 communicates with other devices. For example, the communication unit 110 is directly connected to the LAN 99 and communicates with the airport camera 200, the line-of-sight sensor 300, the aviation information management system 500, etc. The communication unit 110 may be implemented by a communication interface 811.

[0074] The storage unit 120 stores programs and data for the operation of the air traffic control support server 100. The storage unit 120 can be implemented by the storage device 809.

[0075] The control unit 130 provides various functions of the control support server 100. The control unit 130 includes a monitoring target object detection unit 140, a monitoring priority area calculation unit 150, a priority line calculation unit 160, a first incidental information placement unit 170, a second incidental information placement unit 180, and a transmission control unit 190. The control unit 130 can be implemented by a CPU 803, a ROM 805, and a RAM 807.

[0076] The monitored object detection unit 140 detects the position of each aircraft 60 in the captured video based on the captured video of the airport 10 received from the airport camera 200. More specifically, the monitored object detection unit 140 identifies the display position of the image of each aircraft 60 shown in the captured video based on the captured video or the three-dimensional position information of each aircraft 60 acquired from the aviation information management system 500.

[0077] Then, the monitored object detection unit 140 associates, for each aircraft 60, the identification information and flight information corresponding to the aircraft 60 with the position of the aircraft 60, and stores the associated information in the storage unit 120. That is, the monitored object detection unit 140 calculates the on-screen area position of the monitored object detection area A442 shown in Fig. 4 above, and performs processing to associate the on-screen area position of the monitored object detection area A442 with the identification information and flight information of the aircraft shown in the monitored object detection area A442.

[0078] The display position of the image of the aircraft 60 may be determined by object detection processing of each aircraft image. The object detection processing is processing for determining where an object of a certain category is located in an image. Any known method such as YOLO may be used as a specific method for the object detection processing. For example, YOLO is described in Redmon, J. et al., 2016, "You Only Look Once: Unified, real-time object detection," Proceedings of CVPR '16, pp. 779-788.

[0079] Alternatively, the display position of the image of the aircraft 60 may be determined by a process of determining the two-dimensional position of the image of each aircraft 60 by projecting the three-dimensional position of each aircraft 60 onto a two-dimensional airport-captured video image based on the three-dimensional position information of each aircraft 60 acquired from the aviation information management system 500. Note that camera parameter information required for projective transformation of the three-dimensional position into two-dimensional space may be determined in advance using a known camera calibration method. Also, as described above, the air traffic controller 90 may view the airport 10, which is a real space, through a see-through display. At this time, the monitored object detection unit 140 detects the position of each aircraft 60 on the screen A40. More specifically, the monitored object detection unit 140 detects the position of each aircraft 60 on the screen A40 based on the three-dimensional position information of each aircraft 60 acquired from the aviation information management system 500. For example, the position of each aircraft 60 on screen A40 may be determined by a process that obtains the two-dimensional position of each aircraft 60 on screen A40 by projecting the three-dimensional position of each aircraft 60 onto screen A40 based on the three-dimensional position information of each aircraft 60 obtained from the aviation information management system 500.

[0080] Here, we assume a case where a method is used to identify the display position of an image of an aircraft 60 by combining object detection processing and projective transformation. Specifically, based on the three-dimensional position information of each aircraft 60 acquired from the aviation information management system 500, the monitored object detection unit 140 calculates a two-dimensional position by projecting the three-dimensional position onto the airport-captured video using camera parameter information of the airport camera 200. Then, the monitored object detection unit 140 performs object detection processing on the aircraft image around the two-dimensional position on the airport-captured video. The monitored object detection unit 140 performs labeling processing on the detected area of ​​the aircraft image to temporarily assign label information, and then performs processing to link the aircraft's identification information and operation information with the area of ​​the aircraft image to which the label information has been assigned.

[0081] Through the above processing, for example, the monitored object detection area A442A in Figure 4, the operation information of the aircraft 60 which is the monitored object related to the monitored object detection area A442A, and the example of the identification information (for example, call sign) of the aircraft 60, "OKI001", are linked.

[0082] 4, the monitored object detection area A442 is expressed as a "rectangle" circumscribing the area of ​​the aircraft image as an example. However, the shape of the monitored object detection area A442 may be any other polygon that includes the area of ​​the aircraft image, or may be a curved closed area.

[0083] It is also assumed that the above-described object detection process may be difficult. In such cases, the monitored object detection unit 140 may detect, as the position of the monitored object detection area A442, a two-dimensional position obtained by projecting the three-dimensional position information of each aircraft 60 acquired from the aviation information management system 500 onto the airport-captured video. Examples of cases in which the object detection process is difficult include when the aircraft 60 is located far from the airport 10 and is therefore captured in a small size (for example, one pixel) in the airport-captured video, or when the shooting environment is poor due to weather or the like.

[0084] The fact that the aircraft 60 is captured at a small size in the airport-captured video may be detected by the fact that the distance between the airport camera 200 and the aircraft 60 is greater than a threshold, which is calculated based on the three-dimensional relative position information between the airport camera 200 and the aircraft 60. The fact that the shooting environment is poor may be detected by the fact that the shooting time period is a predetermined time period (for example, nighttime), or that the weather at the location where the airport camera 200 is installed is predetermined weather (for example, rainfall, snowfall, etc.).

[0085] The monitoring priority area calculation unit 150 calculates the position of the monitoring priority area of ​​the controller 90A in the captured video image based on information about the gaze point of the controller 90A generated by the above-mentioned line-of-sight sensor 300. More specifically, the monitoring priority area calculation unit 150 calculates the position of the monitoring priority area of ​​the controller 90A in the captured video image A401 (FIG. 4) of the airport 10 based on information about the gaze point of the controller 90A. For example, the monitoring priority area calculation unit 150 calculates the position of the monitoring priority area from information accumulated over a predetermined period of time about the gaze point of the controller 90A.

[0086] The key monitoring area is a part of the captured video of the airport that the air traffic controller places importance on (for example, a part of the runway video), and is a two-dimensional area in the captured video A401 that the controller 90A monitors with a focus, i.e., a two-dimensional area in the captured video A401 where the controller 90A's gaze point is concentrated (for example, when the shape of the captured video A401 (i.e., the shape of the screen A40) is three-dimensional, such as when the display unit of the air traffic control information presentation device 400 is a curved display, the key monitoring area may be a three-dimensional area), and is exemplified as key monitoring area A451 in FIG. 4 above. As mentioned above, the controller 90 may view the airport 10, which is a real space, through a see-through display. At this time, the key monitoring area calculation unit 150 calculates the position of the key monitoring area of ​​the controller 90A on the screen A40. More specifically, the key monitoring area calculation unit 150 calculates the position of the key monitoring area of ​​the controller 90A on the screen A40 based on information about the gaze point of the controller 90A.

[0087] The monitoring priority area may be, for example, a "convex hull" including a group of gaze points within a predetermined time period of the controller 90. In this case, the algorithm for calculating the convex hull may be a divide-and-conquer method, a Graham scan, or any other known method.

[0088] Furthermore, in order to reduce the amount of calculation, the gaze points to be calculated for the convex hull do not need to be all gaze points within a predetermined time. For example, the monitoring priority area calculation unit 150 may divide the screen A40 into any resolution, cumulatively calculate the number of gaze points and gaze dwell time included in each element area of ​​that resolution, and perform filtering processing for the element areas to be calculated for the convex hull based on the cumulative calculation results. For example, the monitoring priority area calculation unit 150 may exclude element areas with the number of gaze points equal to or less than N (N is a natural number) from the convex hull calculation. Note that the conditions and thresholds for the filtering processing may be set in advance by an administrator of the system according to the embodiment of the present invention.

[0089] The priority line calculation unit 160 calculates, as the position of the priority line, the position of a line corresponding to the position of the aircraft 60 detected by the monitored object detection unit 140 and the position of the monitoring priority area A451 calculated by the monitoring priority area calculation unit 150. More specifically, the priority line calculation unit 160 calculates, as the position of the priority line, the position of a line corresponding to the position of the aircraft 60 in the captured video A401 and the position of the monitoring priority area A451 in the captured video A401.

[0090] For example, the priority line calculation unit 160 calculates a center of gravity A461 of the monitored object detection area from the monitored object detection area A442 calculated by the monitored object detection unit 140. The priority line calculation unit 160 also calculates a center of gravity A462 of the monitoring priority area from the monitoring priority area A451 calculated by the monitoring priority area calculation unit 150. The priority line calculation unit 160 then calculates the position of an priority line A463, which is at least a part of a line passing through the two center of gravity positions, in the captured video image A401. As described above, the air traffic controller 90 may view the airport 10, which is a real space, through a transmissive display. In this case, the priority line calculation unit 160 calculates the position of a line corresponding to the position of the aircraft 60 on the screen A40 and the position of the monitoring priority area A451 on the screen A40, as the position of the priority line on the screen A40.

[0091] The center of gravity of the monitored object detection area A442 (e.g., a rectangle) and the center of gravity of the monitoring priority area A451 (e.g., a convex hull polygon) can be calculated using a method in which, for example, a polygon that is the shape of the area is divided into a plurality of triangles, the center of gravity of each triangle is determined, and the center of gravity of each triangle is weighted averaged. However, the centers of gravity of these areas may also be calculated using any other known method. Note that, while the center of gravity point A461 of the monitored object detection area, the center of gravity point A462 of the monitoring priority area, and the priority line A463 are illustrated in FIG. 4 for the purpose of explanation, the center of gravity and the priority line may not be displayed on the screen A40 of the control information presentation device 400 during operation.

[0092] The first incidental information placement unit 170 determines the display position of the first incidental information A471, which is information related to the aircraft 60, at a position corresponding to the position of the aircraft 60 and the position of the monitoring focus area A451. More specifically, the first incidental information placement unit 170 determines the display position of the first incidental information A471 based on the position of the focus line A463 corresponding to the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring focus area.

[0093] 4, as an example, the first incidental information A471A includes aircraft identification information "OKI001," and examples of flight information include spot position "SP01," takeoff position "RWY36," scheduled spot departure time "13:30," and scheduled takeoff time "13:45." However, the information included in the first incidental information A471 may be any information related to the aircraft 60.

[0094] The display position of the first auxiliary information A471 is preferably determined to be a position linked to the image of the aircraft 60 associated with the first auxiliary information A471 (i.e., the monitored object detection area A442). This allows the first auxiliary information A471 and the image of the aircraft 60 associated with the first auxiliary information A471 to be visually linked and displayed, thereby reducing the mental burden on the controller 90A. More specifically, as shown in Fig. 4, the center of gravity A472 of the area of ​​the first auxiliary information may be determined on an emphasis line A463 that is at least a part of a line passing through the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring emphasis area.

[0095] 4, the display position of the first incidental information A471 is preferably determined to be a position close to (including a "circumscribing position") the key monitoring area A451. This reduces the visual search range of the controller 90A for the first incidental information A471 based on the key monitoring area A451, thereby reducing the mental load on the controller 90A.

[0096] The first incidental information placement unit 170 places the first incidental information A471 at the determined display position. More specifically, the first incidental information placement unit 170 places the first incidental information A471 at the position in the captured video A401 that has been determined as the display position of the first incidental information A471. Based on the placement of the first incidental information A471 in this manner, data of the air traffic control assistance information can be generated.

[0097] The second incidental information placement unit 180 determines the display position of the second incidental information A481, which is information related to the aircraft 60, for each aircraft 60 based on the position of the aircraft 60. More specifically, the second incidental information placement unit 180 determines the display position of the second incidental information A481 for each aircraft 60 based on the position of the monitored object detection area A442. For example, it is preferable that the display position of the second incidental information A481 be determined to be a position close to the monitored object detection area A442 (including a "circumscribing position"). This allows the controller 90A or 90B to move their line of sight only a short distance from viewing an image of the aircraft 60 to viewing the second incidental information A481 related to that aircraft 60, thereby reducing the mental load on the controller 90A or 90B.

[0098] 4, as an example, the second incidental information A481A includes information of the same content as the information included in the first incidental information A471A. However, the information included in the second incidental information A481A does not have to be the same as the information included in the first incidental information A471A. The information included in the second incidental information A481A may be any information related to the aircraft 60.

[0099] An emphasis line A483 indicating the link between the second incidental information A481 (or its center of gravity A482) and the image A441 of each aircraft 60 (or its center of gravity A461) may be presented. This makes it possible to more clearly visualize the relationship between the aircraft and the corresponding incidental information. More specifically, the emphasis line A483 may be at least a part of a line passing through the center of gravity A482 of the area of ​​the second incidental information and the center of gravity A461 of the monitored object detection area.

[0100] 5, the center of gravity A482 of the area of ​​the second auxiliary information may be determined on a line passing through the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring focus area. In this case, the priority line A483 and the priority line A463 are arranged parallel to each other. As a result, when the controller 90A (first controller) moves his / her gaze from a position close to the center of gravity A462 of the monitoring focus area to the image A441 of each aircraft 60, the second auxiliary information A481 is located at the end of the extension of the line, which may make it easier to visually recognize the second auxiliary information A481 corresponding to the aircraft 60 together with the aircraft 60. On the other hand, if the second auxiliary information A481 were arranged in a position perpendicular to the priority line A463, the direction of gaze movement would also have to be bent so as to be perpendicular, which may prevent a smooth gaze flow.

[0101] The second incidental information placement unit 180 places the second incidental information A481 at the determined display position. More specifically, the second incidental information placement unit 180 places the second incidental information A481 at the position in the captured video A401 that has been determined as the display position of the second incidental information A481. Based on the placement of the second incidental information A481 in this manner, data of the air traffic control assistance information can be generated.

[0102] The transmission control unit 190 transmits the generated data of the air traffic control support information to the air traffic control information presentation device 400 via the communication unit 110, and causes the air traffic control support information to be presented in an AR format on the screen A40 of the air traffic control information presentation device 400. Specifically, the center of gravity A472 of the area of ​​the first incidental information is on the priority line A463 calculated by the priority line calculation unit 160, and further, the first incidental information A471 is displayed at a position close to (including a "circumscribing position") the monitoring priority area A451 calculated by the monitoring priority area calculation unit 150.

[0103] As described above, the emphasis line A463 does not have to actually be displayed on the screen A40 of the air traffic control information presentation device 400, but it is preferable that the first incidental information A471 and the image A441 of the aircraft 60 are presented on the screen A40 of the air traffic control information presentation device 400 in a manner that indicates a link between them, as illustrated in Fig. 4. In the example of Fig. 5, the manner in which the link is indicated is expressed by displaying on the screen only a line segment that connects the monitored object detection area A442 (for example, the center of gravity A461 of the monitored object detection area) and the first incidental information A471 (for example, the center of gravity A472 of the area of ​​the first incidental information) on a line that passes through the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring focus area.

[0104] In the conventional method of presenting only video images of the airport to the controller, the controller 90 must frequently and widely shift his or her gaze (visual search) between the numerous aircraft images (A441) scattered across the wide screen of the video images of the airport and the paper flight sheet in front of him or her, and furthermore, the workload of memorizing the linking information during such large gaze shifts is imposed. For example, after looking at the paper flight sheet in front of him or on his or her desk (head down) and memorizing the contents, he or she must make a large gaze shift (head up) to the screen to search for the image of the aircraft corresponding to the flight sheet.

[0105] In contrast, in the method of presenting supplementary information that is a feature of the embodiment of the present invention described above, a part of the area on the captured video image of the airport that is important to the controller 90 is designated as a priority monitoring area (A451), and information is presented so that supplementary information related to each aircraft (such as information on the electronic flight sheet) is concentrated in a position near the periphery of the priority monitoring area, so that the range of the controller 90's line of sight search hardly extends beyond the priority monitoring area, and the visual work load can be reduced compared to conventional methods.

[0106] Furthermore, in the system according to an embodiment of the present invention, the incidental information and the aircraft are presented in a linked manner by the priority line (A463), which reduces the burden on the controller of having to memorize information from a paper flight chart and search for the corresponding aircraft on the screen, as in the conventional system. Furthermore, the position of the incidental information automatically changes in response to changes in the display position of the aircraft image on the screen (for example, if an aircraft moves upward on the screen, the linked incidental information also moves upward accordingly). Therefore, even if the controller does not constantly directly monitor the group of aircraft images on the screen, the controller can notice changes in the situation of the monitored object simply by observing the movement of the incidental information near the monitoring priority area. By utilizing the technology according to an embodiment of the present invention, the controller can obtain the benefits described above, for example.

[0107] (2. Description of operation) Next, an example of information processing operation according to this embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 and Fig. 8 are explanatory diagrams showing an example of the operation flow of the air traffic control support system according to this embodiment.

[0108] 7, the airport camera 200 captures video of the airport 10 and transmits the captured video data to the air traffic control support server 100 via the LAN 99 (S1101). The line-of-sight sensor 300 measures the gaze point of the controller 90A and transmits information about the gaze point of the controller 90A to the air traffic control support server 100 via the LAN 99 (S1103). Furthermore, the aviation information management system 500 transmits three-dimensional position information of each aircraft 60 to the air traffic control support server 100 via the LAN 99 (S1105). In addition, the aviation information management system 500 transmits identification information and operation information of each aircraft 60 to the air traffic control support server 100 via the LAN 99.

[0109] In the air traffic control support server 100, the communication unit 110 receives captured video data of the airport 10 from the airport camera 200, receives information on the gaze point of the air traffic controller 90A from the line-of-sight sensor 300, and receives three-dimensional position information, identification information, and operation information of each aircraft 60 from the aviation information management system 500. The monitored object detection unit 140 calculates the area of ​​each aircraft 60 shown in the captured video data as a monitored object detection area A442 based on the captured video data of the airport 10 (S1107).

[0110] Next, the key monitoring area calculation unit 150 calculates the key monitoring area of ​​the controller 90A in the captured video image A401 of the airport 10 based on the information on the gaze point of the controller 90A (S1109).

[0111] Next, the priority line calculation unit 160 calculates a center of gravity A461 of the monitoring target object detection area from the monitoring target object detection area A442. The priority line calculation unit 160 also calculates a center of gravity A462 of the monitoring target area from the monitoring target area A451 calculated by the monitoring target area calculation unit 150. Furthermore, the priority line calculation unit 160 calculates the position in the captured moving image A401 of an priority line A463, which is at least a part of a line passing through the two center of gravity positions (S1111).

[0112] The first incidental information placement unit 170 determines the display position of the first incidental information A471 on an emphasis line A463 according to the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring emphasis area.

[0113] Furthermore, it is preferable that the display position of the first incidental information A471 be determined to be a position adjacent to the key monitoring area A451 (including a "circumscribing position"). This reduces the range of movement of the controller 90A's line of sight to the first incidental information A471 based on the key monitoring area A451, thereby reducing the load on the controller 90A. The first incidental information placement unit 170 places the first incidental information A471 at the determined display position (S1113).

[0114] The second incidental information placement unit 180 determines the display position of the second incidental information A481 to be a position adjacent to the monitored object detection area A442 (including a "circumscribing position"). This reduces the distance that the controller 90A or 90B needs to move their line of sight from viewing an image of the aircraft 60 to viewing the second incidental information A481 related to the aircraft 60, thereby reducing the mental load on the controller 90A or 90B. The second incidental information placement unit 180 places the second incidental information A481 at the determined display position (S1114).

[0115] In this way, data of air traffic control support information can be generated based on the arrangement of the first auxiliary information A471 and the second auxiliary information A481. The transmission control unit 190 transmits the generated data of air traffic control support information to the air traffic control information presentation device 400 via the communication unit 110. In the air traffic control information presentation device 400, the control unit 430 controls the display unit 450 to display the air traffic control support information on the screen A40 (S1115). By displaying the emphasis line A463 on the screen, the first auxiliary information A471 and the image of the aircraft 60 related to the first auxiliary information A471 are visually linked and displayed, eliminating the need to link and memorize the correspondence between the two, thereby reducing the mental burden on the air traffic controller 90A.

[0116] (3. Various Modifications) Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0117] (Variation 1) In the above explanation, the representative points of the monitored object detection area and the monitoring focus area are taken as the centers of gravity (geometric centers) for determining the positions of the priority lines and supplementary information, but the representative points of these areas may be positions other than the centers of gravity.

[0118] Furthermore, for example, if multiple aircraft 60 are present in similar directions as viewed from the center of gravity A462 of the monitoring priority area (i.e., within a range in which the angle formed by connecting the center of gravity of the detected moving images of any two aircraft 60 and the center of gravity of the monitoring priority area falls within a predetermined angle), multiple pieces of incidental information will be displayed on the screen in a concentrated manner in similar positions, and in some cases the display ranges of multiple pieces of incidental information may overlap, impairing the visibility of the information. Therefore, the incidental information may be positioned at a position shifted from the priority line. Alternatively, the incidental information may be positioned at a position shifted away from the monitoring priority area based on a position circumscribing the monitoring priority area.

[0119] More specifically, it is assumed that a plurality of aircraft 60 are present within a range that falls within a predetermined angle with respect to the center of gravity A462 of the monitoring priority area. In such a case, the first auxiliary information placement unit 170 may determine the display position of the first auxiliary information A471 corresponding to at least one aircraft 60 among the plurality of aircraft 60 at a position shifted in a direction away from the first auxiliary information A471 corresponding to the other aircraft 60, or at a position shifted in a direction away from the monitoring priority area A451. This reduces the overlapping range between the pieces of auxiliary information.

[0120] 5, for example, first incidental information A473B is shown in a position circumscribing the monitoring priority area A451, and first incidental information A473C is shown on an emphasis line A463C. However, with respect to the center of gravity A462 of the monitoring priority area as a reference, the angle formed by connecting the center of gravity A461B of the monitored object detection area and the center of gravity A461C of the monitored object detection area falls within a range that falls within a predetermined angle.

[0121] In such a case, the first incidental information placement unit 170 may change the display position of the first incidental information A473B so as to move away from the key monitoring area A451 (i.e., the first incidental information A473B may be changed to the first incidental information A471B). At this time, as shown in Fig. 5, the center of gravity A474B of the area of ​​the first incidental information moves to the center of gravity A472B of the area of ​​the first incidental information.

[0122] In this case, the first incidental information arrangement unit 170 may change the display position of the first incidental information A473C so that the first incidental information A473C is farther from the other first incidental information A473B (here, since the first incidental information A473C is above the first incidental information A473B, the first incidental information A473C may be moved upward) (i.e., the first incidental information A473C may be changed to the first incidental information A471C). Note that if the first incidental information A473C is below the first incidental information A473B, the display position of the first incidental information A473C may be moved downward. At this time, as shown in FIG. 5, the center of gravity A474C of the area of ​​the first incidental information moves to the center of gravity A472C of the area of ​​the first incidental information.

[0123] (Variation 2) The second incidental information placement unit 180 may determine the display position of the second incidental information A481 corresponding to the aircraft 60 only when the air traffic controller 90A or the air traffic controller 90B is looking at the image A441 of the aircraft 60 (the point of gaze is within or near the area of ​​the image A441). In other words, when the air traffic controller 90A or the air traffic controller 90B is not looking at the image A441 of the aircraft 60, the presentation of the second incidental information A481 corresponding to the aircraft 60 on the screen of the air traffic control information presentation device 400 may be stopped.

[0124] As a result, when controller 90A or controller 90B is not looking at image A441 of an aircraft 60, image A441 corresponding to that aircraft 60 is not displayed on the screen, which prevents a large amount of additional information from being displayed on the screen when there are a large number of aircraft 60 taking off or landing, making visual search difficult.

[0125] For example, the second incidental information placement unit 180 may determine whether the controller 90A or 90B is looking at the image of the aircraft 60 based on information about the gaze point of the controller 90A or 90B measured by the line-of-sight sensor 300, based on whether the gaze point of the controller 90B is on the image of the aircraft 60. For example, the second incidental information placement unit 180 may determine whether the controller 90A or 90B is looking at the image of the aircraft 60 based on whether the gaze point of the controller 90A or 90B has stayed within the area of ​​image A441 of the aircraft 60 for a predetermined period of time.

[0126] (Variation 3) In the above description, the presence of the line-of-sight sensor 300 is essential for the operation of the monitoring priority area calculation unit 150. However, the monitoring priority area calculation unit 150 may operate in such a manner that the setting information of the monitoring priority area is input in advance to the system according to the embodiment of the present invention by the air traffic controller 90 or an administrator of the system according to the embodiment of the present invention.

[0127] 4 may be designated by, for example, air traffic controller 90 specifying in advance with an input device (for example, a pointing device such as a mouse) an area of ​​any position or range (for example, a part of a runway image) on captured video image A401 of airport 10. This allows monitoring priority area calculation unit 150 and the system according to the embodiment of the present invention to operate even in a situation or environment where line-of-sight sensor 300 malfunctions and stops working, or where line-of-sight sensor 300 is not provided.

[0128] (Variation 4) In the above description, it has been mainly assumed that the system according to the embodiment of the present invention uses captured video images of the airport. However, instead of captured video images of the airport, the system according to the embodiment of the present invention may use captured still images of the airport. In other words, the system according to the embodiment of the present invention may use any captured images of the airport taken by the airport camera 200.

[0129] (4. Summary) As described above, according to the embodiment of the present invention, when a controller performs airfield control work, an airfield control support system is realized that can reduce the workload of the controller by reducing the vast monitoring range that is required in the conventional airfield control work environment, such as looking directly at each aircraft or looking at a flight sheet on a desk, and the mental load required to link and memorize the status of each aircraft with the information on the flight sheet.

[0130] In addition, according to the present invention, when multiple controllers jointly perform airport control operations using a system according to an embodiment of the present invention, an airport control support system is realized that makes it easier for a second controller, different from the first controller who measures the key monitoring area, to view each aircraft along with the associated information corresponding to it. [Explanation of symbols]

[0131] 100 Control Support Server 110 Communications Department 120 Storage section 130 Control Unit 140 Monitoring object detection unit 150 Monitoring Priority Area Calculation Department 160 Importance line calculation unit 170 First Supplementary Information Placement Section 180 Second Supplementary Information Placement Section 190 Transmission control section 200 Airport Camera 30 Remote Control Room 300 Eye Sensor 400 Control information presentation device 410 Communications Department 420 Storage section 430 Control Unit 450 Display section 500 Aviation Information Management System 60 aircraft 90 Controller

Claims

1. a monitoring target object detection unit that detects the position of a monitoring target object in a captured image or on a screen; a monitoring focus area calculation unit that calculates the position of a monitoring focus area of ​​the user in the captured image or the screen based on information about the user's gaze point generated by a line-of-sight sensor; a first incidental information placement unit that determines a display position of first incidental information, which is information related to the object to be monitored, at a position corresponding to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area; An information processing device comprising:

2. The information processing device includes: an emphasis line calculation unit that calculates the position of an emphasis line that is at least a part of a line that passes through the position of the monitored object and the position of the focused monitoring area; the first auxiliary information placement unit determines a display position of the first auxiliary information based on the position of the emphasis line; The information processing device according to claim 1 .

3. The information processing device includes: a second incidental information placement unit that determines a display position of second incidental information, which is information related to the object to be monitored and different from the first incidental information, at a position near the position of the object to be monitored; The information processing device according to claim 2 .

4. the second incidental information placement unit determines a display position of the second incidental information at a position based on the position of the emphasis line and close to the position of the monitored object; The information processing device according to claim 3 .

5. the second incidental information placement unit determines a display position of the second incidental information based on whether the user's gaze point is within or near an image area of ​​the monitored object in the captured image or the screen; 5. The information processing device according to claim 3.

6. The information processing device includes: a transmission control unit that transmits data in which the first incidental information is arranged at the display position of the first incidental information determined by the first incidental information arrangement unit to a presentation control device via a communication unit; The information processing device according to any one of claims 1 to 5.

7. Detecting the position of a monitored object in a captured image or on a screen; Calculating the position of the user's key monitoring area in the captured image based on information about the user's gaze point generated by an eye-gaze sensor; determining a display position of first incidental information, which is information related to the object to be monitored, at a position according to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area; An information processing method comprising:

8. Computer, a monitoring target object detection unit that detects the position of a monitoring target object in a captured image or on a screen; a monitoring focus area calculation unit that calculates the position of a monitoring focus area of ​​the user in the captured image or the screen based on information about the user's gaze point generated by a line-of-sight sensor; a first incidental information placement unit that determines a display position of first incidental information, which is information related to the object to be monitored, at a position corresponding to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area; A program that causes the information processing device to function as an information processing device having the above.

9. a monitoring target object detection unit that detects the position of a monitoring target object in a captured image or on a screen; a monitoring focus area calculation unit that calculates the position of a monitoring focus area of ​​the user in the captured image or the screen based on information about the user's gaze point generated by a line-of-sight sensor; a first incidental information placement unit that determines a display position of first incidental information, which is information related to the object to be monitored, at a position corresponding to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area; an information processing device comprising: a control unit that controls the display unit so that the first auxiliary information is displayed at the display position, a presentation control device; A system comprising:

Images