Flight robot image display device and image display method

Abstract

To improve visibility of a mark superimposed on a captured image in superimposing and displaying a mark indicating an object detected from the captured image captured by a flight robot.SOLUTION: An image display device comprises: an acquisition unit 24c which acquires a captured image 5 captured by a flight robot 1; an object detection unit 24d which detects an object 6 from the captured image 5; a determination unit 24f which determines a display mode of a mark 7 indicating the detected object 6, according to the characteristic of the captured image 5; and a display unit 21 which displays the captured image 5 having the mark 7 superimposed thereon so as to be displayed in the determined display mode.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display method for a flying robot. [Background technology]

[0002] The following Patent Document 1 describes an imaging device that calculates the reliability that a detected person is a specific person based on a facial image of the detected person and facial feature information of the specific person stored in a database, and changes the display method of the detection frame based on the reliability. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4840978 Summary of the Invention [Problem to be solved by the invention]

[0004] When monitoring images captured by a flying robot, a mark indicating the detected object (for example, a detection frame surrounding the detected object) may be superimposed on the captured image to make it easier for the operator to understand the object detected in the captured image. However, the shooting conditions for flying robots are diverse, and the situations in which objects are detected and the states of the detected objects are also diverse. Therefore, depending on the shooting conditions, the situation in which the object is detected, and the state of the object, there is a risk that the visibility of the mark superimposed on the captured image may be poor. The present invention has been made in consideration of the above-mentioned problems, and aims to improve the visibility of marks superimposed on a captured image when the marks indicating objects detected from the captured image taken by a flying robot are superimposed on the captured image. [Means for solving the problem]

[0005] An image display device for a flying robot according to one embodiment of the present invention comprises an acquisition unit that acquires a captured image taken by the flying robot, an object detection unit that detects an object from the captured image, a determination unit that determines the display format of a mark indicating the detected object according to the characteristics of the captured image, and a display unit that displays the captured image with the mark superimposed so as to be displayed in the display format. [Effects of the Invention]

[0006] According to the present invention, when a mark indicating an object detected from an image captured by a flying robot is superimposed on the captured image, the visibility of the mark superimposed on the captured image can be improved. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing an example of the configuration of a flying robot control system according to an embodiment of the present invention. FIG. [Figure 2] 10(a) and 10(b) are schematic diagrams of a captured image on which a mark indicating an object on the captured image is superimposed. [Figure 3] FIG. 2 is a block diagram illustrating an example of the functional configuration of an autonomous flying robot. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of a management apparatus. [Figure 5] FIG. 2 is a block diagram illustrating an example of a functional configuration of an operation terminal. [Figure 6] 10 is a flowchart illustrating an example of an image display method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments of the present invention shown below are merely examples of devices and methods for embodying the technical concept of the present invention, and the technical concept of the present invention does not limit the structure, arrangement, etc. of the components to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims.

[0009] Hereinafter, the configuration of an embodiment of the present invention will be described with reference to FIG. 1, which shows a schematic configuration of a flying robot control system 100 to which the present invention is applied. (Flying Robot Control System 100) The flying robot control system 100 includes one or more autonomous flying robots 1, an operation terminal 2, and a management device 3. For example, the flying robot control system 100 is used for a wide range of purposes, such as patrolling a predetermined monitoring area, responding to abnormalities in a monitored object, and inspecting the area.

[0010] When the autonomous flying robot 1 performs a patrol flight, the autonomous flying robot 1 is caused to fly autonomously along a flight path set by the management device 3, for example, while detecting the surrounding conditions using a sensor (for example, a camera) provided on the autonomous flying robot 1. A remote operator checks the conditions within the monitored area based on the images taken by the autonomous flying robot 1.

[0011] The autonomous flying robot 1 can also be controlled by an operator manually operating the operation terminal 2. For example, if there is a need to check the situation at a specific location in more detail, the operator can manually control the autonomous flying robot 1 via the operation terminal 2 to check the situation at the specific location in more detail.

[0012] (Outline of flight control method according to embodiment) Next, an example of an image display method according to an embodiment will be outlined with reference to Figures 2(a) and 2(b). A captured image 5 captured by the autonomous flying robot 1 is transmitted to the operation terminal 2. The operation terminal 2 displays the captured image 5 received from the autonomous flying robot 1 on a display device (display unit 21, described below) and presents it to an operator.

[0013] The operation terminal 2 uses image recognition processing to detect objects 6 (moving objects) such as people, vehicles, and animals other than people from the captured image 5. To make it easier for the operator to recognize the objects 6 detected in the captured image 5, the operation terminal 2 displays a mark 7 (for example, a detection frame surrounding the detected object) indicating the detected object 6 superimposed on the captured image 5.

[0014] However, the conditions for capturing images using the autonomous flying robot 1 are diverse, and the circumstances for detecting the object 6 and the state of the detected object 6 are also diverse. Therefore, depending on the capturing conditions, the circumstances for detecting the object 6, and the state of the object 6, there is a risk that the visibility of the mark 7 may be poor. For example, if the hue, lightness, and saturation of the color of the mark 7 are significantly different from the hue, lightness, and saturation of the color of the background of the photographed image 5, the mark 7 will be easily visible, as shown in Figure 2(a). Conversely, if the hue, lightness, and saturation of the color of the mark 7 are close to the hue, lightness, and saturation of the color of the background of the photographed image 5, the mark 7 will blend into the background of the photographed image 5, making the mark 7 difficult to see.

[0015] Therefore, the operation terminal 2 determines the display mode of the mark 7 indicating the detected object 6 according to the characteristics of the captured image 5. This prevents the visibility of the mark 7 superimposed on the captured image 5 from becoming poor depending on the shooting conditions of the captured image 5, the detection status of the object 6, and the state of the object 6, thereby improving the visibility of the mark 7.

[0016] Next, the autonomous flying robot 1, the operation terminal 2, and the management device 3 will be described. (Autonomous Flying Robot 1) 3 is a block diagram of an example of the functional configuration of an autonomous flying robot. The autonomous flying robot 1 is, for example, a small unmanned helicopter of the hexa-rotor or quad-rotor type, such as a multicopter, drone, or UAV (Unmanned Aerial Vehicle). Note that the present invention is not limited to small unmanned helicopters of the hexa-rotor or quad-rotor type, and can also be applied to small unmanned helicopters of the single-rotor type. The autonomous flying robot 1 receives information related to the flight path of the autonomous flying robot 1 (path information, which will be described later) from the operation terminal 2.

[0017] When the autonomous flying robot 1 is made to fly autonomously, the autonomous flying robot 1 flies automatically according to predetermined flight conditions. The predetermined flight conditions may be, for example, a flight path specified by route information received from the management device 3. In the autonomous flight mode, the autonomous flying robot 1 flies along the flight path specified by the management device 3. A target position may be set as a predetermined flight condition. In this case, the autonomous flying robot 1 searches for a flight path toward the target position while avoiding obstacles, and flies along the searched flight path.

[0018] The autonomous flying robot 1 includes a position and attitude sensor 10 , an imaging device 11 , a communication unit 12 , a control unit 13 , a memory unit 14 , a motor 15 , and a rotor 16 . The position and attitude sensor 10 acquires the current position and attitude of the autonomous flying robot 1. The position and attitude sensor 10 includes, for example, a receiver that receives radio waves (navigation signals) transmitted from navigation satellites (artificial satellites) such as the Global Navigation Satellite System (GNSS), an acceleration sensor that measures acceleration, an electronic compass that measures orientation, and a gyro sensor that measures angular velocity.

[0019] For example, the receiver of the position and attitude sensor 10 receives navigation signals transmitted from a plurality of navigation satellites and outputs them to the control unit 13 , and the electronic compass and gyro sensor output measurement signals to the control unit 13 . Instead of a receiver, other known sensors, such as a laser scanner and a barometric pressure sensor, may be used to obtain information for obtaining the current position and attitude using known prior art techniques.

[0020] The imaging device 11 is an imaging device (e.g., a camera) for capturing an image seen from the position of the autonomous flying robot 1. For example, the imaging device 11 may be a camera with a fixed angle of view and a fixed optical axis direction relative to the front-to-rear direction of the autonomous flying robot 1, or a camera with a variable angle of view and a variable optical axis direction (e.g., a PTZ camera).

[0021] The communication unit 12 is a communication module for communication between the operation terminal 2 and the autonomous flying robot 1, and between the management device 3 and the autonomous flying robot 1. The storage unit 14 is an information storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), or an HDD (Hard Disk Drive). The storage unit 14 stores various programs and various data, and inputs and outputs this information to and from the control unit 13. The various data includes information used in processing by the control unit 13, such as position and orientation information 14a and path information 14b.

[0022] The position and attitude information 14a is a history of positions and attitudes in which the current position and attitude of the autonomous flying robot 1 acquired by the position and attitude sensor 10 are cyclically stored a predetermined number of times. Here, the most recent position and attitude in the history are particularly referred to as the current position and current attitude, and other positions and attitudes may be referred to as past positions and past attitudes. The route information 14b is information about the flight route along which the autonomous flying robot 1 is scheduled to move. Specifically, it is a coordinate sequence on the flight route. The route information 14b is received from the management device 3.

[0023] The control unit 13 is a computer equipped with a processor such as a CPU (Central Processing Unit), and peripheral devices such as a ROM, a RAM, etc. The processor of the control unit 13 executes a computer program stored in the storage unit 14 to realize the functions of the control unit 13 described below. The control unit 13 functions as a position / attitude calculation unit 13a, a flight control unit 13b, a route search unit 13c, a flight information transmission unit 13d, a captured image transmission unit 13e, and the like.

[0024] The position / attitude calculation unit 13a calculates the current position and attitude of the autonomous flying robot 1 in a three-dimensional movement area (e.g., flight space) from the output of the position / attitude sensor 10, and stores it in the memory unit 14 as position / attitude information 14a. For example, the position / attitude calculation unit 13a obtains latitude, longitude, and altitude from the navigation signal output by the position / attitude sensor 10, and converts them into a position in the coordinate system of the moving area using a pre-stored conversion rule.

[0025] Furthermore, the position / orientation calculation unit 13a obtains the current orientation in the coordinate system of the moving area from the measurement signals of the acceleration sensor and gyro sensor output by the position / orientation sensor 10. The direction in the coordinate system of the moving area may be obtained from the measurement signal of the electronic compass output by the position and orientation sensor 10, and the current orientation may be calculated using further measurement signals from other sensors. As described above, the position and attitude sensor 10 and the position and attitude calculation unit 13a (control unit 13) work together to detect the current position and attitude of the autonomous flying robot 1.

[0026] The flight control unit 13b controls the flight behavior of the autonomous flying robot 1 by controlling the rotation speed of the motors 15. The autonomous flying robot 1 is equipped with four or six rotors 16 and motors 15 each having a rotation shaft connected to each of these rotors 16. Each motor 15 is connected to the control unit 13 and receives instructions on its rotation speed from the flight control unit 13b. The independent rotation of the four or six rotors 16 causes the autonomous flying robot 1 to generate acceleration in any direction.

[0027] When autonomous flying robot 1 is caused to fly autonomously, flight control unit 13b automatically flies autonomous flying robot 1 in accordance with predetermined flight conditions. For example, if a flight path is given as a predetermined flight condition, flight control unit 13b references route information 14b and position / attitude information 14a and controls the rotation speed of motor 15 so that autonomous flying robot 1 moves following the flight path described in route information 14b. Specifically, flight control unit 13b controls the rotation speed of motor 15 so as to reduce the error between the position (coordinates) at the current time described in route information 14b and the current position described in position / attitude information 14a.

[0028] For example, when a target position is given as a predetermined flight condition, the rotation speed of the motor 15 is controlled so that the robot moves along a flight path toward the target position while avoiding obstacles. The path search unit 13c searches for a flight path toward the target position while avoiding obstacles. When the autonomous flying robot 1 is manually controlled, the flight control unit 13b controls the rotation speed of the motor 15 based on control information received from the operation terminal 2. The operation terminal 2 may transmit, for example, information on target accelerations in three axial directions (forward / backward, left / right, and up / down) to be generated by the autonomous flying robot 1 as control information.

[0029] The flight information transmission unit 13d transmits flight information related to the flight of the autonomous flying robot 1 to the operation terminal 2 and the management device 3 via the communication unit 12. For example, the flight information may include information such as the current position, current attitude, altitude, and speed of the autonomous flying robot 1. The captured image transmission unit 13e transmits the captured image captured by the imaging device 11 to the operation terminal 2 via the communication unit 12.

[0030] (Management device 3) The management device 3 is installed in a predetermined location (for example, a monitoring area, a movement area of ​​the autonomous flying robot 1, or the premises of a business operator that operates the autonomous flying robot 1). The management device 3 calculates a flight path from the current position received from the autonomous flying robot 1 to the target position to which the autonomous flying robot 1 will fly, and transmits this to the autonomous flying robot 1.

[0031] 4 is a block diagram showing an example of the functional configuration of the management device 3. The management device 3 includes a communication unit 30, a control unit 31, and a storage unit 32. The communication unit 30 is a communication module for communication between the autonomous flying robot 1 and the management device 3, and between the operation terminal 2 and the management device 3.

[0032] The memory unit 32 is an information storage device such as a ROM, RAM, or HDD. The memory unit 32 stores various programs and various data, and inputs and outputs this information to and from the control unit 31. The various data includes information used in processing by the control unit 31, such as map information 32a of the monitoring area and its surrounding area, route information 32b, and flight information 32c received from the autonomous flying robot 1. The control unit 31 is a computer equipped with a processor such as a CPU, and peripheral devices such as a ROM, a RAM, etc. The processor of the control unit 31 executes a computer program stored in the storage unit 32 to realize the functions of the control unit 31 described below.

[0033] The control unit 31 functions as a flight route setting unit 31a and a flight information acquisition unit 31b. The flight path setting unit 31a searches for a flight path for moving the autonomous flying robot 1 from the starting position of the autonomous flying robot 1's movement (for example, the current position of the autonomous flying robot 1) to a target position set as the destination of the autonomous flying robot 1, and stores information about the searched flight path in the memory unit 32 as path information 32b.

[0034] The flight path setting unit 31a transmits information about the searched flight path to the autonomous flying robot 1 via the communication unit 30. The autonomous flying robot 1 stores the received flight path information as path information 14b. The flight information acquisition unit 31b acquires flight information 32c of the autonomous flying robot 1 transmitted from the autonomous flying robot 1. The flight information acquisition unit 31b stores the acquired flight information 32c in the memory unit 32.

[0035] (Operation terminal 2) The operation terminal 2 is a terminal device used by an operator to control the autonomous flying robot 1 and to monitor captured images sent from the autonomous flying robot 1. For example, the operation terminal 2 may be a remote control device (a so-called "radio transmitter") or tablet terminal that can be carried by the operator, or may be a personal computer fixed to a monitoring table or the like. The operation terminal 2 is an example of an "image display device" as defined in the claims.

[0036] 5 is a block diagram showing an example of the functional configuration of the operation terminal 2. The operation terminal 2 includes a communication unit 20, a display unit 21, an operation unit 22, a control unit 24, and a storage unit 25. The communication unit 20 is a communication module for communicating between the autonomous flying robot 1 and the operation terminal 2, and between the management device 3 and the operation terminal 2. For example, the communication unit 20 receives captured images transmitted from the autonomous flying robot 1.

[0037] The display unit 21 is a user interface device that displays visual information provided to the operator from the operation terminal 2. For example, the operation terminal 2 displays a captured image received from the autonomous flying robot 1 on the display unit 21 and presents it to the operator. The operation unit 22 is a user interface that accepts an operation input from an operator to the operation terminal 2. The operation unit 22 may be, for example, a touch panel provided on the display screen of the display unit 21 on which a graphical user interface (GUI) is displayed, or may have physical operation members such as buttons, dials, and sliders.

[0038] The memory unit 25 is an information storage device such as a ROM, RAM, or HDD. The memory unit 25 stores various programs and various data, and inputs and outputs this information to and from the control unit 24. The various data includes information used in processing by the control unit 24, such as map information 25a of the monitoring area and its surrounding area, and flight information 25b received from the autonomous flying robot 1.

[0039] The control unit 24 is a computer equipped with a processor such as a CPU, and peripheral devices such as a ROM, a RAM, etc. The processor of the control unit 24 executes a computer program stored in the storage unit 25 to realize the functions of the control unit 24 described below. The control unit 24 functions as a control information generation unit 24a, a flight information acquisition unit 24b, a captured image reception unit 24c, an object detection unit 24d, a color information acquisition unit 24e, a mode determination unit 24f, and an image output unit 24g.

[0040] When the autonomous flying robot 1 is manually controlled, the control information generator 24a generates control information for controlling the flight behavior of the autonomous flying robot 1 based on the operator's operation input to the operation unit 22. The control information generator 24a transmits the control information to the autonomous flying robot 1 via the communication unit 20. The flight information acquisition unit 24b acquires flight information 25b of the autonomous flying robot 1 transmitted from the autonomous flying robot 1. The flight information acquisition unit 24b stores the acquired flight information 25b in the memory unit 25.

[0041] The captured image receiving unit 24c acquires the captured image received from the autonomous flying robot 1. Below, the object detection unit 24d, the state determination unit 24f, and the image output unit 24g will be described using the captured image 5 shown in FIG. 2(a) as an example of a captured image received from the autonomous flying robot 1. The object detection unit 24d detects an object 6 from the captured image 5 by image recognition processing. The object 6 detected by the object detection unit 24d may be, for example, a specific type of moving object or stationary object that the operator should be aware of. For example, the object 6 may be a person, an animal other than a person, or an object other than a person (for example, a vehicle).

[0042] The color information acquisition unit 24e acquires color information for a specific region of the captured image 5. The color information includes at least one of hue, lightness, and saturation. For example, an RGB image is acquired as the captured image 5, and converted from the RGB color space to the HSV color space. The color information acquisition unit 24e acquires the hue value H (0° to 360°), lightness S (0% to 100%), and saturation V (0% to 100%) in the HSV space for each pixel constituting the HSV image, and creates a histogram for each HSV value. The maximum value of each histogram is then found as the representative value of the HSV, and this is used as color information for the captured image 5. Note that instead of calculating the representative value, the average value of the HSV values ​​for each pixel may be calculated and used as color information for the captured image 5. Furthermore, in addition to the example of acquiring color information for the entire captured image 5, color information for the captured image 5 may be acquired for a predetermined range around the image position of the object 6 in the captured image 5 (for example, a range of a predetermined distance with the image position of the object 6 as the center position), or color information may be acquired for the image position of the object 6 in the captured image 5 (i.e., object 6). Furthermore, texture information in the captured image (for example, shadow areas, patterns, people's clothing, etc.) may be obtained using a known method. Note that color information and texture information are examples of "background information" as defined in the claims.

[0043] The mode determining unit 24f determines the display mode of the mark 7 that indicates the object 6 detected by the object detecting unit 24d on the captured image 5. For example, the mark 7 may be a detection frame that surrounds the object 6, or a picture, figure, symbol, etc. that is added near the image position of the object 6. Furthermore, the display mode of the mark 7 may be, for example, the size, color (color information), or shape of the mark 7, or the thickness of the line segments that make up the detection frame, figure, or symbol.

[0044] The mode determining unit 24f determines the display mode of the mark 7 according to the characteristics of the captured image 5. The characteristic of the captured image 5 may be, for example, color information (e.g., hue, brightness, saturation) of the captured image 5. For example, the mode determining unit 24f may determine the color (e.g., hue, brightness, saturation) of the mark 7, which is the display mode of the mark 7, in accordance with the color information of the captured image 5. The manner determination unit 24f may determine, as the color of the mark 7, a color that differs from the color of the captured image 5 in at least one of hue, lightness, and saturation. For example, the manner determination unit 24f may determine the color of the mark 7 so that at least one of hue, lightness, and saturation differs between the color of the captured image 5 and the color of the mark 7 by a threshold or more. For example, the manner determination unit 24f may determine, as the color of the mark 7, an opposite or complementary color to the color of the captured image 5, and determine, as the color of the mark 7, a color that differs by a predetermined angle (e.g., 180°) on the color wheel from the hue value identified as the color information of the captured image 5. Furthermore, the manner determination unit 24f may determine, as the color of the mark, a color that differs from the lightness or saturation of the color of the captured image 5 by a threshold or more (e.g., 30%) from the lightness or saturation identified as the color information of the captured image 5. Furthermore, the display mode determining unit 24f may determine the display mode of the mark 7 according to the texture information. For example, if the background of the captured image 5 is a checkered pattern, the mark 7 may be displayed in a shape other than a rectangle to improve the visibility of the mark 7.

[0045] For example, the manner determination unit 24f may determine the color of the mark 7 based on color information in a predetermined range around the image position of the object 6 in the captured image 5 (for example, a range of a predetermined distance centered on the image position of the object 6). Also, for example, taking into account the case where the object 6 has moved, the color of the mark 7 may be determined based on the overall color information of the captured image 5. For example, the color of the mark 7 may be determined based on the average values ​​of the overall hue, brightness, and saturation of the captured image 5.

[0046] Alternatively, for example, the display mode of the mark 7 may be determined based on the similarity between color information in a predetermined range around the image position of the object 6 in the captured image 5 and the color information of the object 6. The similarity of the color information is determined to be high when the difference between at least one of the HSV values ​​is less than a threshold value, for example. When the similarity is high, i.e., when the object 6 and the background are similar in color, there is a risk that the object 6 will blend in with the background and become difficult to see, so it is necessary to further improve the visibility of the mark 7. Therefore, when the similarity is high, the line segments constituting the mark 7 may be made thicker, or the size of the mark 7 may be made larger than the size of the mark 7 determined based on the size of the object 6 in the captured image 5.

[0047] Furthermore, for example, the characteristic of the captured image 5 may be the number or density of the objects 6 detected in the captured image 5. That is, the mode determining unit 24f may determine the display mode of the marks 7 according to the number or density of the objects 6 detected in the captured image 5. When the total number of objects 6 detected in the captured image 5 is large, or when the density of objects 6 is high due to the objects 6 concentrating in a specific area, there is a risk that the visibility of the marks 7 will be poor if the lines constituting the detection frame, figure, or symbol, which is the mark 7, are too thick.

[0048] For this reason, for example, the manner determining unit 24f may determine the thickness of the line segments constituting the mark 7 in accordance with the number or density of objects 6 detected in the captured image 5. For example, the manner determining unit 24f may make the line segments constituting the mark 7 thinner when the total number of objects 6 detected in the captured image 5 is large compared to when it is small. Furthermore, for example, the manner determining unit 24f may divide the captured image 5 into a plurality of areas, and make the line segments constituting the mark 7 thinner when the number of objects 6 detected in each divided area is large (i.e., when the density is high) compared to when it is small (i.e., when the density is low).

[0049] Furthermore, for example, the characteristic of the captured image 5 may be color information of the image of the object 6 in the captured image 5. That is, the manner determining unit 24f may determine the color of the mark 7, which is the display manner of the mark 7, according to the color information of the image of the object 6. If the color of the image of the object 6 and the color of the mark 7 are similar, the mark 7 may blend in with the image of the object 6, reducing visibility.

[0050] For this reason, the manner determining unit 24f may determine, as the color of the mark 7, a color that differs in at least one of hue, lightness, and saturation from the color of the image of the object 6. For example, the manner determining unit 24f may determine the color of the mark 7 so that at least one of hue, lightness, and saturation differs between the color of the image of the object 6 and the color of the mark 7 by a threshold or more. For example, the manner determining unit 24f may determine, as the color of the mark 7, an opposite color or complementary color to the color of the image of the object 6.

[0051] For example, the characteristics of the captured image 5 may be flight information of the autonomous flying robot 1 when the captured image 5 was captured. In other words, the mode determination unit 24f may determine the display mode of the mark 7 according to the flight information of the autonomous flying robot 1 when the captured image 5 was captured. For example, if the autonomous flying robot 1 is at a high altitude when the photographed image 5 is captured, the object 6 is likely to appear small in the photographed image 5. As a result, if the size of the mark 7 (e.g., detection frame) is determined according to the size of the object 6 in the photographed image 5, there is a risk that the mark 7 will be difficult to see (i.e., the position of the object 6 will be difficult to grasp).

[0052] For this reason, the mode determination unit 24f may use altitude information of the autonomous flying robot 1 as flight information of the autonomous flying robot 1. The mode determination unit 24f may determine the size of the mark 7, which is the display mode of the mark 7, depending on the altitude of the autonomous flying robot 1. For example, when the altitude of the autonomous flying robot 1 is above a threshold, the size of the mark 7 may be made larger than the size of the mark 7 determined based on the size of the object 6 on the captured image 5, in order to make it easier to grasp the position of the object 6.

[0053] For example, if the autonomous flying robot 1 is traveling at a high speed when the captured image 5 is captured, the position of the object 6 is likely to deviate from the mark 7 (for example, the detection frame). For this reason, the mode determination unit 24f may use the speed of the autonomous flying robot 1 as flight information of the autonomous flying robot 1. The mode determination unit 24f may determine the size of the mark 7, which is the display mode of the mark 7, depending on the speed of the autonomous flying robot 1. For example, when the speed of the autonomous flying robot 1 is high, the mark 7 may be made larger than when the speed is low, so that the position of the object 6 is less likely to deviate from the mark 7 (for example, so that it is easier to fit within the detection frame). For example, when the speed of the autonomous flying robot 1 is equal to or higher than a threshold, the size of the mark 7 may be made larger than the size of the mark 7 determined depending on the size of the object 6 in the captured image 5.

[0054] Furthermore, for example, it may be possible to identify approximate color information of the captured image 5 based on the flying area of ​​the autonomous flying robot 1 when the captured image 5 was captured. For example, if the flying area is the ocean, the color of the captured image 5 can be identified as approximately blue; if the flying area is a golf course, the color can be identified as approximately green; and if the flying area is farmland, the color can be identified as approximately green during the busy farming season and approximately brown during the off-season.

[0055] For this reason, the mode determination unit 24f may use the flight area of ​​the autonomous flying robot 1 as flight information for the autonomous flying robot 1. The mode determination unit 24f may obtain information about the flight area of ​​the autonomous flying robot 1 based on the current position of the autonomous flying robot 1 and map information 25a. Alternatively, for example, the operator may determine the flight area based on the flight plan of the autonomous flying robot 1 and input the flight area by operating the operation unit 22.

[0056] The mode determination unit 24f may determine the color of the mark 7, which is the display mode of the mark 7, depending on the flight area of ​​the autonomous flying robot 1. For example, color information for the mark 7 depending on the flight area or type of flight area may be set in advance and stored in the memory unit 25. The color information for the mark 7 depending on the flight area or type of flight area may be determined based on the background color of the captured image 5 captured in the respective flight area. The method for determining the color of the mark 7 relative to the background color of the captured image captured in the flight area may be the same as the method for determining the color of the mark 7 relative to the color information of the captured image 5 or the image of the object 6 described above.

[0057] The manner determining unit 24f may read color information that is set in advance in accordance with the acquired flight area from the storage unit 25 and determine the color of the mark 7 as the color. The image output unit 24g displays the captured image 5 with the mark 7 superimposed on it on the display unit 21. At this time, the image output unit 24g sets the display mode of the mark 7 displayed on the display unit 21 to the display mode determined by the mode determination unit 24f.

[0058] In the above explanation, an embodiment has been exemplified in which the operation terminal 2 determines the display format of the mark 7 and superimposes the mark 7 on the captured image 5, but some or all of this processing may also be performed by the autonomous flying robot 1 or the management device 3. For example, some or all of the functions of the object detection unit 24d, the mode determination unit 24f, and the image output unit 24g of the operation terminal 2 may be executed by the control unit 13 of the autonomous flying robot 1 or the control unit 31 of the management device 3. The management device 3 may include a display device (not shown) that displays the captured image 5 with the mark 7 superimposed thereon.

[0059] In other words, some or all of the processing by which the operating terminal 2 determines the display mode of the mark 7 and superimposes the mark 7 on the captured image 5 may be performed in a distributed manner by the autonomous flying robot 1, the operating terminal 2, and the management device 3, and the communication unit 12 of the autonomous flying robot 1, the communication unit 20 of the operating terminal 2, and the communication unit 30 of the management device 3 may send and receive information necessary for the processing to and from each other.

[0060] (operation) FIG. 6 is a flowchart illustrating an example of an image display method according to the embodiment. In step S1, the captured image receiving unit 24c acquires the captured image 5 received from the autonomous flying robot 1. In step S2, the object detection unit 24d detects the object 6 from the captured image 5 by image recognition processing.

[0061] In step S3, the manner determining unit 24f determines the display manner of the mark 7 indicating the object 6 on the photographed image 5 in accordance with the characteristics of the photographed image 5. In step S4, the image output unit 24g displays the captured image 5 on the display unit 21 with the mark 7 superimposed thereon in the display format determined in step S3. The process then ends.

[0062] (Effects of the embodiment) (1) The image display device includes a captured image receiving unit 24c that acquires captured images taken by the autonomous flying robot 1, an object detection unit 24d that detects objects from the captured images, a display mode determination unit 24f that determines the display mode of a mark indicating the detected object according to the characteristics of the captured image, and a display unit 21 that displays the captured image with the mark superimposed so that it is displayed in the determined display mode. This prevents the visibility of the mark superimposed on the captured image from becoming poor due to the shooting conditions of the image captured by the autonomous flying robot 1, the object detection situation, and the state of the object, thereby improving the visibility of the mark.

[0063] (2) The mode determining unit 24f may determine the color of the mark, which is the display mode, according to color information in the captured image. This prevents the visibility of the mark from being reduced due to the color of the mark being similar to that of the captured image, thereby improving the visibility of the mark.

[0064] (3) The mode determining unit 24f may determine the display mode according to the number or density of detected objects, or color information of the detected objects in the captured image. This makes it possible to prevent the visibility of the mark from becoming poor depending on the state of detection of the object in the captured image, and improve the visibility of the mark.

[0065] (4) The mode determination unit 24f may determine the display mode depending on the flight information of the autonomous flying robot 1. This prevents the visibility of the marks superimposed on the captured image according to the flight information of the autonomous flying robot 1 from becoming poor, thereby improving the visibility of the marks.

[0066] The image display device for a flying robot according to one embodiment of the present invention can contribute to solving social issues such as a declining labor force, long working hours, etc. Furthermore, the image display device according to one embodiment of the present invention can contribute to achieving Goal 9 of the Sustainable Development Goals (SDGs) adopted by the United Nations, "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience." [Explanation of symbols]

[0067] 1...autonomous flying robot, 2...operation terminal, 3...management device, 5...captured image, 6...object, 7...mark, 10...position and attitude sensor, 11...imaging device, 12, 20, 30...communication unit, 13, 24, 31...control unit, 13a...position and attitude calculation unit, 13b...flight control unit, 13c...route search unit, 13d...flight information transmission unit, 13e...captured image transmission unit, 14, 25, 32...storage unit, 14a...position and attitude Information, 14b, 32b...route information, 15...motor, 16...rotor, 21...display unit, 22...operation unit, 24a...control information generation unit, 24b, 31b...flight information acquisition unit, 24c...captured image receiving unit, 24d...object detection unit, 24f...mode determination unit, 24g...image output unit, 25a, 32a...map information, 25b, 32c...flight information, 31a...flight path setting unit, 100...flying robot control system

Claims

1. an acquisition unit that acquires images captured by the flying robot; an object detection unit that detects an object from the captured image; a determination unit that determines a display mode of a mark indicating the detected object in accordance with characteristics of the captured image; a display unit that displays the captured image with the mark superimposed thereon so as to be displayed in the display mode; An image display device for a flying robot, comprising:

2. The image display device for a flying robot according to claim 1, wherein the determination unit determines the display mode according to background information in the captured image.

3. The image display device for a flying robot according to claim 1, characterized in that the determination unit determines the display mode depending on the number, density, or color information of the objects detected in the captured image.

4. The image display device for a flying robot according to claim 2, characterized in that the determination unit determines color information of the mark, which is the display mode, according to color information of the area surrounding the detected object within the area of ​​the captured image.

5. The image display device for a flying robot according to claim 2, characterized in that the determination unit determines the display mode according to the degree of similarity between color information of the area surrounding the detected object in the captured image and color information of the detected object.

6. The flying robot image display device according to claim 1, wherein the determination unit determines the display mode in accordance with flight information of the flying robot.

7. The flying robot image display device described in claim 6, characterized in that the determination unit uses altitude information of the flying robot as flight information of the flying robot, and makes the size of the mark, which is the display mode, larger when the altitude of the flying robot is high than when it is low.

8. A process of acquiring images taken by the flying robot; A process of detecting an object from the captured image; A process of determining a display mode of a mark indicating the detected object according to characteristics of the captured image; a process of displaying the captured image with the mark superimposed thereon so as to be displayed in the display mode; 2. An image display method comprising: causing at least one computer to execute the above steps.

Images