Flight robot image display device and image display method

Abstract

To reduce a labor for monitoring an image captured by a flight robot and flight information of the flight robot.SOLUTION: A flight robot image display device comprises: a storage unit 25 which stores a map 42 including a flight area of a flight robot 1; an acquisition unit 24b which acquires from the flight robot 1 a captured image 41 captured by the flight robot 1 and information on the present position of the flight robot 1; and a display unit 21 which displays the captured image 41 and overlays and displays a map displaying the present position of the flight robot on the captured image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display method for a flying robot. [Background technology]

[0002] The following Patent Document 1 describes a display control device that displays thumbnail images of images captured by a camera on a planar map image displayed on a display screen. The display position of the thumbnail image is set to a position on the map that corresponds to the shooting position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4468794 Summary of the Invention [Problem to be solved by the invention]

[0004] When monitoring images captured by a flying robot or the flight position of the flying robot, if the display area for this information is allocated independently on a single display screen, the display area for each may become small, reducing visibility.On the other hand, if this information is displayed on multiple display screens, the visibility may be impaired and the burden on the operator may increase. The present invention has been made in consideration of the above problems, and aims to reduce the effort required to monitor images taken by a flying robot and flight information of the flying robot. [Means for solving the problem]

[0005] An image display device for a flying robot according to one embodiment of the present invention comprises a memory unit that stores a map including the flying area of ​​the flying robot, an acquisition unit that acquires from the flying robot an image taken by the flying robot and information on the current location of the flying robot, and a display unit that displays the captured image and overlays a map showing the current location of the flying robot on the captured image. [Effects of the Invention]

[0006] According to the present invention, the effort required to monitor images taken by a flying robot and flight information of the flying robot can be reduced. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing an example of the configuration of a flying robot control system according to an embodiment of the present invention. FIG. [Figure 2] FIG. 2 is a block diagram illustrating an example of the functional configuration of an autonomous flying robot. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of a management apparatus. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of an operation terminal. [Figure 5] FIG. 2 is a schematic diagram illustrating an example of a graphical user interface displayed on a display unit of an operation terminal. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of a map image displayed on a display unit of an operation terminal. [Figure 7] 10(a) and 10(b) are explanatory diagrams of a first modified example of the display mode of the overlay display. [Figure 8] 10(a) and 10(b) are explanatory diagrams of a second modified example of the display mode of the overlay display. [Figure 9] FIG. 10 is an explanatory diagram of a third modified example of the display mode of the overlay display. [Figure 10] 1 is a flowchart of an image display method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments of the present invention shown below are merely examples of devices and methods for embodying the technical concept of the present invention, and the technical concept of the present invention does not limit the structure, arrangement, etc. of the components to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims.

[0009] <System configuration> Hereinafter, the configuration of an embodiment of the present invention will be described with reference to FIG. 1, which shows a schematic configuration of a flying robot control system 100 to which the present invention is applied. (Flying Robot Control System 100) The flying robot control system 100 includes one or more autonomous flying robots 1, an operation terminal 2, and a management device 3. For example, the flying robot control system 100 is used for a wide range of purposes, such as patrolling a predetermined monitoring area, responding to abnormalities in a monitored object, and inspecting the area.

[0010] When the autonomous flying robot 1 performs a patrol flight, the autonomous flying robot 1 is caused to fly autonomously along a flight path set by the management device 3, for example, while detecting the surrounding conditions using a sensor (for example, a camera) provided on the autonomous flying robot 1. A remote operator checks the situation within the monitored area based on the detection results of the autonomous flying robot 1's sensor (for example, images captured by a camera).

[0011] The autonomous flying robot 1 can also be controlled by an operator manually operating the operation terminal 2. For example, if there is a need to check the situation at a specific location in more detail, the operator can manually control the autonomous flying robot 1 via the operation terminal 2 to check the situation at the specific location in more detail.

[0012] (Autonomous Flying Robot 1) The autonomous flying robot 1 is, for example, a small unmanned helicopter of the hexa-rotor or quad-rotor type, such as a multicopter, drone, UAV (Unmanned Aerial Vehicle), etc. Note that the present invention is not limited to small unmanned helicopters of the hexa-rotor or quad-rotor type, and can be similarly applied to small unmanned helicopters of the single-rotor type. The autonomous flying robot 1 receives information related to the flight path of the autonomous flying robot 1 (path information, which will be described later) from the management device 3.

[0013] When the autonomous flying robot 1 is made to fly autonomously, the autonomous flying robot 1 flies automatically according to predetermined flight conditions. The predetermined flight condition may be, for example, a flight path specified by route information received from the management device 3. In the autonomous flight mode, the autonomous flying robot 1 flies along the flight path specified by the management device 3. A target position may also be set as the predetermined flight condition. In this case, the autonomous flying robot 1 searches for a flight path toward the target position while avoiding obstacles, and flies along the searched flight path.

[0014] 2, the autonomous flying robot 1 includes a position and attitude sensor 10, an external sensor 11, a communication unit 12, a control unit 13, a memory unit 14, a motor 15, and a rotor 16. The position and attitude sensor 10 acquires the current position and attitude of the autonomous flying robot 1. The position and attitude sensor 10 includes, for example, a receiver that receives radio waves (navigation signals) transmitted from navigation satellites (artificial satellites) such as the Global Navigation Satellite System (GNSS), an acceleration sensor that measures acceleration, an electronic compass that measures orientation, and a gyro sensor that measures angular velocity.

[0015] For example, the receiver of the position and attitude sensor 10 receives navigation signals transmitted from a plurality of navigation satellites and outputs them to the control unit 13 , and the electronic compass and gyro sensor output measurement signals to the control unit 13 . Instead of a receiver, other known sensors, such as a laser scanner and a barometric pressure sensor, may be used to obtain the current position and attitude information using known prior art techniques.

[0016] The external sensor 11 is a sensor for detecting the surrounding conditions of the autonomous flying robot 1. For example, the external sensor 11 may be a camera, a laser sensor, or an infrared sensor. The communication unit 12 is a communication module for communication between the operation terminal 2 and the autonomous flying robot 1, and between the management device 3 and the autonomous flying robot 1. The storage unit 14 is an information storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), or an HDD (Hard Disk Drive). The storage unit 14 stores various programs and various data, and inputs and outputs this information to and from the control unit 13. The various data includes information used in processing by the control unit 13, such as position and orientation information 14a and path information 14b.

[0017] The position and attitude information 14a is a history of positions and attitudes in which the current position and attitude of the autonomous flying robot 1 acquired by the position and attitude sensor 10 are cyclically stored a predetermined number of times. Here, the most recent position and attitude in the history are particularly referred to as the current position and current attitude, and other positions and attitudes may be referred to as past positions and past attitudes. The route information 14b is information about the flight route along which the autonomous flying robot 1 is scheduled to move. Specifically, it is a sequence of coordinates on the flight route. The route information 14b is received from the management device 3.

[0018] The control unit 13 is a computer equipped with a processor such as a CPU (Central Processing Unit), and peripheral devices such as a ROM, a RAM, etc. The processor of the control unit 13 executes a computer program stored in the storage unit 14 to realize the functions of the control unit 13 described below. The control unit 13 functions as a position / attitude calculation unit 13a, a flight control unit 13b, a sensor information transmission unit 13c, a flight information transmission unit 13d, a flight status transmission unit 13e, and the like.

[0019] The position / attitude calculation unit 13a calculates the current position and attitude of the autonomous flying robot 1 in a three-dimensional movement area (e.g., flight space) from the output of the position / attitude sensor 10, and stores it in the memory unit 14 as position / attitude information 14a. For example, the position / attitude calculation unit 13a obtains latitude, longitude, and altitude from the navigation signal output by the position / attitude sensor 10, and converts them into a position in the coordinate system of the moving area using a pre-stored conversion rule.

[0020] Furthermore, the position / orientation calculation unit 13a obtains the current orientation in the coordinate system of the moving area from the measurement signals of the acceleration sensor and gyro sensor output by the position / orientation sensor 10. The direction in the coordinate system of the moving area may be obtained from the measurement signal of the electronic compass output by the position and orientation sensor 10, and the current orientation may be calculated using further measurement signals from other sensors. As described above, the position and attitude sensor 10 and the position and attitude calculation unit 13a (control unit 13) work together to detect the current position and attitude of the autonomous flying robot 1.

[0021] The flight control unit 13b controls the flight behavior of the autonomous flying robot 1 by controlling the rotation speed of the motors 15. The autonomous flying robot 1 is equipped with four or six rotors 16 and motors 15 each having a rotation shaft connected to each of these rotors 16. Each motor 15 is connected to the control unit 13 and receives instructions on its rotation speed from the flight control unit 13b. The independent rotation of the four or six rotors 16 causes the autonomous flying robot 1 to generate acceleration in any direction.

[0022] When autonomous flying robot 1 is caused to fly autonomously, flight control unit 13b automatically flies autonomous flying robot 1 in accordance with predetermined flight conditions. For example, if a flight path is given as a predetermined flight condition, flight control unit 13b references route information 14b and position / attitude information 14a and controls the rotation speed of motor 15 so that autonomous flying robot 1 moves following the flight path described in route information 14b. Specifically, flight control unit 13b controls the rotation speed of motor 15 so as to reduce the error between the position (coordinates) at the current time described in route information 14b and the current position described in position / attitude information 14a. For example, if a target position is given as a predetermined flight condition, a flight path toward the target position while avoiding obstacles is searched for, and the rotational speed of the motor 15 is controlled so that the aircraft moves along the searched flight path.

[0023] When the autonomous flying robot 1 is manually controlled, the flight control unit 13b controls the rotation speed of the motor 15 based on control information received from the operation terminal 2. The operation terminal 2 may transmit, for example, information on target accelerations in three axial directions (forward / backward, left / right, and up / down) to be generated by the autonomous flying robot 1 as control information. The sensor information transmitter 13c transmits sensor information about the situation around the autonomous flying robot 1 obtained by the external sensor 11 to the operation terminal 2 and the management device 3 via the communication unit 12. The sensor information may be, for example, an optical image from a camera, a range image from a laser sensor, or an infrared image from an infrared sensor.

[0024] The flight information transmission unit 13d transmits flight information related to the flight of the autonomous flying robot 1 to the operation terminal 2 and the management device 3 via the communication unit 12. For example, the flight information may include information such as the current position of the autonomous flying robot 1, the trajectory of the flight path, the altitude, the shooting direction of the external sensor 11, and the like.

[0025] The flight status transmission unit 13e transmits flight status information regarding the flight status of the autonomous flying robot 1 to the operation terminal 2 and the management device 3 via the communication unit 12. For example, the flight status information may include information about the flight environment of the autonomous flying robot 1 and information about the aircraft status of the autonomous flying robot 1 itself.

[0026] For example, the flight environment may include the wind speed, weather (rain, snow, etc.), and lighting conditions in the area where the autonomous flying robot 1 is flying, information on the speed of the autonomous flying robot 1, information on the altitude of the autonomous flying robot 1, and attributes of the area in which the autonomous flying robot 1 is flying (manned, unmanned, public road, private property, narrow area, etc.). For example, the airframe status of the autonomous flying robot 1 itself may include the battery status of the autonomous flying robot 1 and abnormal flight (such as large shaking).

[0027] (Management device 3) The management device 3 is installed at a predetermined location (for example, a monitoring area, a movement area of ​​the autonomous flying robot 1, or the premises of a business operator operating the autonomous flying robot 1), determines the target position to which the autonomous flying robot 1 will fly, and based on the current position received from the autonomous flying robot 1, calculates a flight path from the current position to the target position and transmits it to the autonomous flying robot 1. 3 is a block diagram showing an example of the functional configuration of the management device 3. The management device 3 includes a communication unit 30, a control unit 31, and a storage unit 32.

[0028] The communication unit 30 is a communication module for communication between the autonomous flying robot 1 and the management device 3, and between the operation terminal 2 and the management device 3. The memory unit 32 is an information storage device such as a ROM, RAM, or HDD. The memory unit 32 stores various programs and various data, and inputs and outputs this information to and from the control unit 31. The various data includes information used in processing by the control unit 31, such as map information 32a of the monitoring area and its surrounding area, route information 32b, and flight information 32c and flight status information 32d received from the autonomous flying robot 1.

[0029] The control unit 31 is a computer equipped with a processor such as a CPU, and peripheral devices such as a ROM, a RAM, etc. The processor of the control unit 31 executes a computer program stored in the storage unit 32 to realize the functions of the control unit 31 described below. The control unit 31 functions as a flight path setting unit 31a, a flight information acquisition unit 31b, and a flight state acquisition unit 31c.

[0030] The flight path setting unit 31a searches for a flight path along which the autonomous flying robot 1 will move from the start position of the autonomous flying robot 1's movement (for example, the current position of the autonomous flying robot 1) to a specified target position, and stores information about the searched flight path as path information 32b in the memory unit 32. The flight path setting unit 31a also transmits the information about the searched flight path to the autonomous flying robot 1 via the communication unit 30. The autonomous flying robot 1 stores the received flight path information as path information 14b.

[0031] The flight information acquisition unit 31b acquires flight information 32c of the autonomous flying robot 1 transmitted from the autonomous flying robot 1. The flight information acquisition unit 31b stores the acquired flight information 32c in the memory unit 32. The flight status acquisition unit 31c acquires flight status information 32d transmitted from the autonomous flying robot 1. The flight status acquisition unit 31c stores the acquired flight status information 32d in the memory unit 32.

[0032] (Operation terminal 2) The operation terminal 2 is a terminal device used by an operator to control the autonomous flying robot 1 and to monitor sensor information sent from the autonomous flying robot 1. For example, the operation terminal 2 may be a remote control device (a so-called "radio transmitter") or tablet terminal that can be carried by the operator, or may be a personal computer fixed to a monitoring table or the like. The operation terminal 2 is an example of an "image display device" as defined in the claims. 4 is a block diagram showing an example of the functional configuration of the operation terminal 2. The operation terminal 2 includes a communication unit 20, a display unit 21, an operation unit 22, a control unit 24, and a storage unit 25.

[0033] The communication unit 20 is a communication module for communication between the autonomous flying robot 1 and the operation terminal 2, and between the management device 3 and the operation terminal 2. The display unit 21 is a user interface device that displays visual information provided from the operation terminal 2 to the operator.

[0034] The operation unit 22 is a user interface that accepts an operation input from an operator to the operation terminal 2. The operation unit 22 may be, for example, a touch panel provided on the display screen of the display unit 21 on which a graphical user interface (GUI) is displayed, or may have physical operation members such as buttons, dials, and sliders.

[0035] The memory unit 25 is an information storage device such as a ROM, RAM, or HDD. The memory unit 25 stores various programs and various data, and inputs and outputs this information to and from the control unit 24. The various data includes, for example, map information 25a of the monitoring area and its surrounding area, and information used in processing by the control unit 24, such as flight information 25b and flight status information 25c received from the autonomous flying robot 1.

[0036] The control unit 24 is a computer equipped with a processor such as a CPU, and peripheral devices such as a ROM, a RAM, etc. The processor of the control unit 24 executes a computer program stored in the storage unit 25 to realize the functions of the control unit 24 described below. The control unit 24 functions as a control information generation unit 24a, a flight information acquisition unit 24b, a sensor information receiving unit 24c, a flight status acquisition unit 24d, a display control unit 24e, a monitoring importance determination unit 24f, a flight purpose acquisition unit 24g, and an object detection unit 24h.

[0037] When the autonomous flying robot 1 is manually controlled, the control information generator 24a generates control information for controlling the flight behavior of the autonomous flying robot 1 based on the operator's operation input to the operation unit 22. The control information generator 24a transmits the control information to the autonomous flying robot 1 via the communication unit 20. The flight information acquisition unit 24b acquires flight information 25b of the autonomous flying robot 1 transmitted from the autonomous flying robot 1. The flight information acquisition unit 24b stores the acquired flight information 25b in the memory unit 25. The flight information acquisition unit 24b is an example of the "acquisition unit" described in the claims.

[0038] The sensor information receiving unit 24c receives sensor information (optical images from a camera, distance images from a laser sensor, and infrared images from an infrared sensor) from the external sensor 11 transmitted from the autonomous flying robot 1. The sensor information receiving unit 24c is an example of the "acquisition unit" described in the claims. The flight state acquisition unit 24d acquires flight state information 25c of the autonomous flying robot 1 transmitted from the autonomous flying robot 1. The flight information acquisition unit 24b stores the acquired flight state information 25c in the memory unit 25.

[0039] The display control unit 24e controls the display of the GUI on the display unit 21. Fig. 5 is a schematic diagram of an example of the GUI displayed on the display unit 21 of the operation terminal 2. The GUI 40 includes a captured image display area, a map image display area, and a steering operation interface unit 43. The captured image display area is an area that displays captured images 41 captured by the autonomous flying robot 1. The captured images 41 captured by the autonomous flying robot 1 may include, for example, optical images captured by a camera, distance images captured by a laser sensor, infrared images captured by an infrared sensor, and the like. The map display area is an area that displays a map image 42 that includes the flight area of ​​the autonomous flying robot 1. The display control unit 24e reads out map information 25a of the flight area of ​​the autonomous flying robot 1 from the storage unit 25, and displays the map image 42 of the read map information 25a in the map display area.

[0040] 6 is a schematic diagram of an example of a map image 42 of map information 25a displayed in the map display area. Map information 25a includes information about the flight area, such as building information about buildings 50 in the flight area, facility information about other facilities 51 (e.g., parking lots), and obstacle information. The map information 25a may include height information of buildings 50, facilities 51, obstacles, etc. The display control unit 24e may express the height of these objects by the display mode (e.g., color, brightness, etc.) of the buildings 50, facilities 51, obstacles, etc. displayed on the map image 42, or may display characters or symbols indicating the height near the figures representing the buildings 50, facilities 51, obstacles, etc.

[0041] The map information 25a may also include information on the location of the hangar 59 (charging station) and takeoff and landing point 58a of the autonomous flying robot 1, the emergency landing point 58b, the monitoring (inspection) location 57, and the like. Furthermore, if the autonomous flying robot 1 is used for patrol, the map information 25a may include information on the patrol route 56.

[0042] Furthermore, the display control unit 24e displays in the map display area the map image 42 including the flight information 25b acquired from the autonomous flying robot 1 by the flight information acquisition unit 24b. In the example of the map image 42 in Fig. 6, the current position 52 of the autonomous flying robot 1, the trajectory 53 of the flight path, the altitude and speed 54, and the shooting direction 55 of the captured image 41 are displayed.

[0043] See Fig. 5. The display control unit 24e overlays the map image 42 displayed in the map display area on the photographed image 41 displayed in the photographed image display area, making the overlapping portion of the map image 42 and the photographed image 41 transparent. For example, the transmittance (transparency) of the map image 42 overlaid on the photographed image 41 is set to a value greater than 0. This allows the operator to see both the map image 42 and the photographed image 41 in the overlapping portion of the map image 42 and the photographed image 41.

[0044] As a result, even when the map image 42 and the photographed image 41 are displayed on a single display screen, a sufficiently large display area can be secured for each image. This makes it possible to avoid a decrease in visibility due to a small display area. Furthermore, by displaying the map image 42 and the photographed image 41 on a single display screen, the visibility is improved and an increase in the burden on the operator can be avoided.

[0045] The control operation interface unit 43 is an operation area that accepts manual operation of the flight behavior of the autonomous flying robot 1 when the operation mode of the autonomous flying robot 1 is in manual flight mode. For example, the operator can manually control the flight behavior (horizontal movement, vertical movement, turning) of the autonomous flying robot 1 by pressing the arrow buttons displayed on the control operation interface unit 43.

[0046] The display control unit 24e may determine the display mode of the overlay display of the map image 42 and the photographed image 41 according to the monitoring importance of the photographed image 41 to be displayed in the photographed image display area. In this specification, the "monitoring importance" refers to the degree to which the photographed image 41 needs to be monitored carefully. For example, the monitoring importance determination unit 24f may determine the monitoring importance based on the flight status information 25c. The monitoring importance determination unit 24f is an example of the "acquisition unit" described in the claims.

[0047] For example, the monitoring importance determination unit 24f may calculate the monitoring importance based on the flight status information 25c as an indicator of whether the situation is such that the captured images 41 taken by the autonomous flying robot 1 should be monitored more carefully, such as a situation where the flight of the autonomous flying robot 1 is highly dangerous. For example, a higher monitoring importance level may be calculated for flights in strong winds or bad weather, flights at low altitudes, flights over populated areas or public roads, etc., since the risks during flight are higher than in other cases. Also, in the case of responding to an abnormality, the need to check the captured images 41 is high, so a higher monitoring importance level may be calculated.

[0048] For example, the monitoring importance determination unit 24f may calculate the monitoring importance by scoring multiple indicators (wind speed, weather, brightness, speed, altitude, area attributes, aircraft condition, etc.), and target cases where the monitoring importance is above a threshold, or may target cases where certain conditions are met, such as when the wind speed is determined to meet the conditions for strong winds, as being of high monitoring importance.

[0049] In such a situation where the monitoring importance is high, the monitoring importance determination unit 24f determines a display format that makes the captured image 41 easier to see, so that the operator can monitor the captured image 41 more carefully and take action as necessary (e.g., manual operation, emergency landing instructions, etc.). For example, the monitoring importance determination unit 24f may determine the display mode as the transmittance (transparency) of the map image 42 overlaid on the photographed image 41, the display area (occupancy area) where the map image 42 is overlaid on the photographed image 41, the occupancy rate of the map image 42 shown in the photographed image 41, the scale of the map image 42, or the display position where the map image 42 is overlaid on the photographed image 41.

[0050] 7(a) and 7(b) are explanatory diagrams of a first modified example of the display mode of the overlay display. Fig. 7(a) shows an example of a standard display mode when the monitoring importance is relatively low, and Fig. 7(b) shows an example of a display mode when the monitoring importance is relatively high (a display mode in which the captured image 41 is easy to see). For example, when the monitoring importance is relatively low, the display area (occupied area) for overlaying the map image 42 on the captured image 41 can be made relatively large, and the map image 42 can be overlaid near the center of the captured image 41, thereby making the map image 42 easier to see.

[0051] On the other hand, when the monitoring importance is relatively high, the display area (occupied area) for overlaying the map image 42 on the captured image 41 can be made relatively small, and the map image 42 can be overlaid near the corner of the captured image 41, thereby making the map image 42 easier to see. For example, the monitoring importance determination unit 24f may lower the transparency of the map image 42 overlaid on the captured image 41 when the monitoring importance is relatively low, and may set the transparency of the map image 42 low when the monitoring importance is relatively high.

[0052] For example, the monitoring importance determination unit 24f may determine the monitoring importance according to the flight purpose of the autonomous flying robot 1. For example, the flight purpose of the autonomous flying robot 1 may include a patrol task, a return task, an emergency return task, a tracking task, a detection and movement task, and a designated point movement task. The patrol task involves flying along a predetermined patrol route, taking photographs at checkpoints, and determining whether there are any abnormalities at the checkpoints (suspicious people, suspicious vehicles, suspicious objects, overheating or cracks in equipment, etc.).

[0053] The return task is a task that searches for a return route to the hangar / takeoff point based on a return instruction from the operator when another task is completed or interrupted, and automatically lands at the hangar or takeoff point. The emergency return task is a task that searches for a return route to the hangar / takeoff point based on a return instruction from the operator in the event of an emergency such as aircraft trouble, and automatically lands at the hangar or takeoff point.

[0054] The tracking task is a task in which the autonomous flying robot 1 flies to track a target (person, car, etc.) based on an instruction to select the target from a monitoring console on which a captured image 41 captured by the autonomous flying robot 1 is displayed. The detection movement task is a task in which, when a sensor installed in the monitored area detects an object or a fixed camera detects an object, the robot flies (or waits to fly) to the position of the detected sensor or fixed camera. The designated point movement task is a task to fly to a position designated by an operation terminal or a management terminal.

[0055] The flight purpose acquisition unit 24g acquires information about the flight purpose of the autonomous flying robot 1 from the management device 3. Alternatively, it may acquire information about the flight purpose entered by an operator operating the operation unit 22. The flight purpose acquisition unit 24g is an example of the "acquisition unit" described in the claims. The monitoring importance determination unit 24f may determine the monitoring importance based on the flight purpose of the autonomous flying robot 1 acquired by the flight purpose acquisition unit 24g. For example, if a tracking task is set as the flight objective, a higher monitoring importance may be determined because it is important for the operator to confirm the target in the captured image 41. Alternatively, if a return task is set, a lower monitoring importance may be determined because it is not important for the operator to confirm the captured image 41.

[0056] For example, the monitoring importance determination unit 24f may set the monitoring importance for the tracking task, the detection movement task, and the emergency return task to the highest level of "high," set the monitoring importance for the return task to the lowest level of "low," and set the monitoring importance for the patrol task and the designated point movement task to the intermediate level of "medium."

[0057] In addition, when a detection target object (hereinafter sometimes referred to as "target object") is detected from the photographed image 41, the display control unit 24e may determine the display mode of the overlay display of the map image 42 and the photographed image 41 according to the detection result of the detection target. The object detection unit 24h detects a target object from the captured image 41. For example, the object detection unit 24h determines a moving object such as a person or a vehicle as the target object. The display control unit 24e displays the map image 42 at a position that does not overlap with the position of the detected target object.

[0058] The processing of the object detection unit 24h may be performed by the management device 3. For example, the control unit 24 of the management device 3 may function as a sensor information receiving unit that receives sensor information of the external sensor 11 transmitted from the autonomous flying robot 1 (i.e., captured image 41 captured by the autonomous flying robot 1), and an object detection unit that detects a target object from the captured image 41. The control unit 24 may transmit position information of the position of the detected target object to the operation terminal 2.

[0059] 8(a) and 8(b) are explanatory diagrams of a third modified example of the display mode of the overlay display. Fig. 8(a) shows an example of the state before the display mode of the overlay display is changed when a person is detected as the target object 60 in the captured image 41. Typically, when the autonomous flying robot 1 photographs the target object 60, it is controlled so that the target object 60 is positioned at the center of the angle of view of the photographed image 41. For this reason, in a standard display mode in which a relatively large map image 42 is overlaid near the center of the photographed image 41, the map image 42 is overlaid on top of the target object 60, which may impair the visibility of the target object 60. 8(b), the display control unit 24e displays the map image 42 in one of the four corners of the photographed image 41. The display control unit 24e may also display the map image 42 in a position where the target object 60 cannot move, such as in the area of ​​a building 44.

[0060] 9 is an explanatory diagram of a third modification of the display mode of the overlay display. The display control unit 24e may take into consideration the moving direction of the target object 60 and may prevent the map image 42 from being displayed in the moving direction of the target object 60. An arrow 61 indicates the moving direction of the target object 60. For example, if the target object 60 is moving in the direction of the bottom right of the captured image 41, there is a risk that the target object 60 will overlap the map image 42 if the map image 42 is displayed in the bottom right corner of the captured image 41 as shown in Figure 8(b). Therefore, the display control unit 24e displays the map image 42 at the corner in the direction opposite to the moving direction 61 of the target object 60 (the upper left corner).

[0061] In the above description, an embodiment in which the function of overlaying the map image 42 and the photographed image 41 is performed by the operation terminal 2 has been exemplified, but part or all of the processing of the function of overlaying the map image 42 on the photographed image 41 may be performed by the management device 3. The control unit 24 of the management device 3 may execute part or all of the functions of the sensor information receiving unit 24c, the display control unit 24e, the flight purpose acquisition unit 24g, the monitoring importance determination unit 24f, and the object detection unit 24h of the operation terminal 2, for example. In this case, the management device 3 may transmit a display image generated by overlaying the map image 42 on the captured image 41 to the operation terminal 2, and the operation terminal 2 may display the display image received from the management device 3 on the display unit 21. The management device 3 may also include a display device (not shown) that displays the map image 42 as an overlay on the captured image 41.

[0062] (operation) FIG. 10 is a flowchart of an image display method according to an embodiment. In step S1, the display control unit 24e of the operation terminal 2 reads out map information 25a including the flight area of ​​the autonomous flying robot 1 from the storage unit 25. In step S2, the sensor information receiving unit 24c acquires from the autonomous flying robot 1 the images captured by the autonomous flying robot 1.

[0063] In step S3, the flight information acquisition unit 24b acquires flight information 25b of the autonomous flying robot 1 transmitted from the autonomous flying robot 1. In step S4, the display control unit 24e overlays the map image of the map information 25a displaying the flight information 25b on the captured image on the display unit 21. The process then ends.

[0064] (Effects of the embodiment) (1) The image display device includes a memory unit 25 that stores a map including the flight area of ​​the autonomous flying robot 1, a sensor information receiving unit 24c and a flight information acquiring unit 24b that acquire images captured by the autonomous flying robot 1 and information on the current location of the autonomous flying robot 1 from the autonomous flying robot 1, and a display unit 21 that overlays a map showing the current location of the autonomous flying robot 1 on the captured image, making the overlapping portion of the map and the captured image transparent.

[0065] This allows a sufficiently large display area to be secured for each image even when the map and the captured image are displayed on a single display screen. This prevents a decrease in visibility due to a small display area. Furthermore, displaying the map and the captured image on a single display screen improves the visibility and prevents an increase in the burden on the operator.

[0066] (2) The display unit 21 may determine the display mode in the overlay display depending on the monitoring importance of the captured image. This allows the operator to change the display mode so that the captured image 41 is easy to view in situations where the operator needs to monitor the captured image more carefully. (3) The image display device may include a flight status acquisition unit 24d that acquires information about the flight status of the autonomous flying robot 1. The display unit 21 may determine the display format for the overlay display depending on the flight status. This allows the display format to be changed so that the captured image 41 is easy to see, for example, in situations where there is a high risk during the flight of the autonomous flying robot 1 and it is highly necessary to check the captured image.

[0067] (4) The display unit 21 may determine the display mode of the overlay display depending on the flight purpose of the autonomous flying robot 1. This allows the display mode to be changed so that the captured image 41 is easy to see when the flight purpose requires the operator to check the captured image 41. (5) The image display device may include an object detection unit 24h that detects an object from the captured image. The display unit may determine the display format for the overlay display based on the detection result by the object detection unit 24h. This prevents the overlay display map from interfering with monitoring of an object detected in the captured image.

[0068] The image display device for a flying robot according to one embodiment of the present invention can contribute to solving social issues such as a declining labor force, long working hours, etc. Furthermore, the image display device according to one embodiment of the present invention can contribute to achieving Goal 9 of the Sustainable Development Goals (SDGs) adopted by the United Nations, "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience." [Explanation of symbols]

[0069] 1...autonomous flying robot, 2...operation terminal, 3...management device, 10...position and attitude sensor, 11...external sensor, 12...communication unit, 13...control unit, 13a...position and attitude calculation unit, 13b...flight control unit, 13c...sensor information transmission unit, 13d...flight information transmission unit, 13e...flight status transmission unit, 14...memory unit, 14a...position and attitude information, 14b...route information, 15...motor, 16...rotor, 20...communication unit, 21...display unit, 22...operation unit, 24...control unit, 24a...control information generation unit, 24b...flight information acquisition unit, 24c...sensor information reception unit, 24d... Flight status acquisition unit, 24e...display control unit, 24f...monitoring importance determination unit, 24g...flight purpose acquisition unit, 24h...object detection unit, 25...memory unit, 25a...map information, 25b...flight information, 25c...flight status information, 30...communication unit, 31...control unit, 31a...flight path setting unit, 31b...flight information acquisition unit, 31c...flight status acquisition unit, 32...memory unit, 32a...map information, 32b...route information, 32c...flight information, 32d...flight status information, 41...captured image, 42...map image, 43...pilot operation interface unit, 100...flying robot control system

Claims

1. a storage unit that stores a map including a flight area of ​​the flying robot; an acquisition unit that acquires from the flying robot an image captured by the flying robot and information on the current position of the flying robot; a display unit that displays the captured image and overlays the map showing the current position of the flying robot on the captured image; An image display device for a flying robot, comprising:

2. the acquisition unit acquires a monitoring importance level of the captured image, The image display device for a flying robot according to claim 1, wherein the display unit determines a display mode of the map in the overlay display in accordance with the monitoring importance level.

3. The acquisition unit acquires information about the flight state of the flying robot, The image display device for a flying robot according to claim 1, wherein the display unit determines a display mode of the map in the overlay display depending on the flight state.

4. The acquisition unit acquires a flight purpose of the flying robot, The image display device for a flying robot according to claim 1, wherein the display unit determines a display mode of the map in the overlay display according to the flight purpose.

5. an object detection unit that detects an object from the captured image; The image display device for a flying robot according to claim 1, wherein the display unit determines a display mode of the map in the overlay display in accordance with a detection result by the object detection unit.

6. The image display device for a flying robot according to claim 2 , wherein the display unit determines at least one of a transparency, a display position, and a display size of the map as the display mode.

7. A process of reading a map including a flight area of ​​the flying robot from a storage device that stores the map; A process of acquiring images taken by the flying robot and current position information of the flying robot from the flying robot; a process of displaying the photographed image and overlaying the map showing the current position of the flying robot on the photographed image; 2. An image display method comprising: causing at least one computer to execute the above steps.

Images