Display control apparatus and control method thereof

Abstract

To perform information display capable of improving operability of a drone mounted with an imaging device.SOLUTION: The display control device includes a video acquisition unit that acquires a first captured video acquired by a first imaging unit mounted on the drone and used for navigation control and a second captured video acquired by a second imaging unit mounted on the drone and used for a purpose other than the navigation control, a state acquisition unit that acquires a state of the drone, and a control unit that controls a display mode when the first captured video and the second captured video are displayed on a display unit based on the state acquired by the state acquisition unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to display control of a plurality of captured images. [Background technology]

[0002] Conventionally, some flying devices (drones) that can move wirelessly are equipped with an imaging device that provides the captured image to a display device. The operator operates the drone while viewing the captured image displayed on the display device. Some display devices are configured to not only display images used for operation but also various information to encourage the operator to understand the drone's status and the surrounding environment. For example, Patent Document 1 proposes a display system that displays images and information images based on video data transmitted from an unmanned mobile work device equipped with a stereo camera. Specifically, the technology discloses a technique for improving workability by displaying stereo images and information images related to the work of the unmanned mobile work device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-164223 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the display system disclosed in Patent Document 1, multiple types of data are displayed within a limited display range, reducing visibility for the operator. For example, as the number of types of data to be displayed increases, the display is reduced, which can lead to the operator overlooking the data and reducing the operability of the drone.

[0005] The present invention has been made in consideration of such problems, and aims to provide a technology for displaying information that can improve the operability of a drone equipped with an imaging device. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, a display control device according to the present invention has the following configuration. an image acquisition means for acquiring a first captured image acquired by a first imaging unit mounted on the drone and used for navigation control of the drone, and a second captured image acquired by a second imaging unit mounted on the drone and used for purposes other than the navigation control; A status acquisition means for acquiring a status of the drone; a control means for controlling a display mode when the first captured image and the second captured image are displayed on a display unit based on the state obtained by the state obtaining means; Equipped with. [Effects of the Invention]

[0007] According to the present invention, a technology can be provided for displaying information that can improve the operability of a drone equipped with an imaging device. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating the overall configuration of a system and the functional configuration of each device. [Figure 2] FIG. 1 is a diagram illustrating a hardware configuration of a drone. [Figure 3] FIG. 2 is a diagram illustrating a hardware configuration of a display device. [Figure 4] FIG. 1 is a diagram illustrating an example of the appearance of a drone. [Figure 5] FIG. 1 is a diagram illustrating an example of the appearance of a display device. [Figure 6] FIG. 2 is a diagram illustrating display control in a display device. [Figure 7] 10A and 10B are diagrams illustrating other display control in the display device. [Figure 8] 4 is a flowchart of display control in the first embodiment. [Figure 9] FIG. 2 is a diagram illustrating display control in a display device. [Figure 10] 10 is a flowchart of display control in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) As a first embodiment of a control device according to the present invention, a display device that controls the display format of two images acquired from a drone having two imaging units will be described below as an example.

[0011] <System configuration and device configuration> 1 is a diagram showing the overall configuration of the system and the functional configuration of each device. The system 10 includes a drone 100 and a display device 200.

[0012] The drone 100 has a first imaging unit 102, a first image processing unit 103, a first imaging control unit 104, a second imaging unit 105, a second image processing unit 106, and a second imaging control unit 107. The drone 100 also has a control unit 101, a wireless communication unit 108, a drive unit 109, a position acquisition unit 110, a speed acquisition unit 111, an attitude acquisition unit 112, a distance acquisition unit 113, and a memory unit 114.

[0013] The first imaging unit 102 and the second imaging unit 105 are composed of an imaging optical system and an imaging element. The imaging optical system has a zoom lens, a focus lens, and an aperture mechanism, and focuses light from the subject onto the light receiving surface of the imaging element. The zoom lens moves in the optical axis direction, making it possible to change the imaging magnification (angle of view). The focus lens moves in the optical axis direction, making it possible to adjust the focus. The aperture mechanism can adjust the amount of light passing through the optical system. Each lens and aperture mechanism has a driving unit, and is controlled by the first imaging control unit 104 and the second imaging control unit 107.

[0014] The image sensor has a semiconductor element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. The image sensor photoelectrically converts the subject image from light incident from the imaging optical system to generate pixel data, which is an analog signal. The image sensor can amplify the signal according to a set signal amplification factor (analog gain). The pixel data output from the first image capturing unit 102 and the second image capturing unit 105 is sent to the first image processing unit 103 and the second image processing unit 106, where it is processed.

[0015] In the first image processing unit 103 and the second image processing unit 106, pixel data is converted into digital signals by A / D conversion. The converted digital signals are converted into image data through correction processes such as black level correction, gamma curve adjustment, noise reduction, and white balance correction, as well as development processes. Data compression processes such as JPEG may also be performed.

[0016] The wireless communication unit 108 transmits the image data output by the first image processing unit 103 and the second image processing unit 106 to the display device 200 via the network 115. The wireless communication unit 108 also transmits and receives various information obtained by the position acquisition unit 110, the speed acquisition unit 111, the attitude acquisition unit 112, and the distance acquisition unit 113. The wireless communication unit 108 also transmits and receives control information for each imaging device by the first imaging control unit 104 and the second imaging control unit 107, and operation information input from the operation input unit 124.

[0017] The drive unit 109 is a mechanical part including a motor and propellers that enables navigation and attitude control of the drone 100. The drone 100 can move (fly) based on instructions in three directions (forward / backward, left / right, and up / down), for example. The drive unit 109 is controlled by the control unit 101 based on operation information from the operation input unit 124 received by the wireless communication unit 108, and various information obtained by the position acquisition unit 110, speed acquisition unit 111, attitude acquisition unit 112, and distance acquisition unit 113.

[0018] The position acquisition unit 110 receives and processes GPS (Global Positioning System) signals to identify the position of the drone 100 and the direction of movement of the drone 100. The speed acquisition unit 111 processes digital data output from an acceleration sensor and the like to calculate the acceleration and speed of the drone 100. The attitude acquisition unit 112 processes digital data output from an angular velocity sensor and the like to calculate changes in the attitude (rotation and direction) of the drone 100. The distance acquisition unit 113 processes data output from a distance sensor / depth sensor such as Lidar (Light Detection and Ranging), millimeter-wave radar, or ultrasonic sensor to calculate the surrounding environment of the drone 100 (distance to surrounding obstacles). Note that distance information may also be calculated from an evaluation value of focus obtained by phase-difference AF (AUTO FOCUS).

[0019] The storage unit 114 can save and read out video data output by the first imaging control unit 104 and the second image processing unit 106. It is also used as a storage area for programs executed by the control unit 101 (described later), a storage area for various parameters, and a work area during program execution.

[0020] The control unit 101 comprehensively controls each component of the drone 100 described above, and sets various parameters, controls flight, and issues instructions for sending and receiving data.

[0021] The display device 200 includes a control unit 120 , a wireless communication unit 121 , a display unit 122 , a display control unit 123 , an operation input unit 124 , a storage unit 125 , and an I / F unit 126 .

[0022] The display unit 122 is controlled by the display control unit 123, and displays images captured by each imaging device and various information. The display control unit 123 controls the display on the display unit 122. For example, the display control unit 123 selects image data and information to be displayed on the display unit 122, controls changes to the display format such as superimposed display or split display, and controls changes to the display area of ​​each image and information. As will be described in detail later, the display control unit 123 changes the display form (display format, display area) of the image and information displayed on the display unit 122 based on information obtained by the position acquisition unit 110, the velocity acquisition unit 111, the attitude acquisition unit 112, and the distance acquisition unit 113, or instructions from the operator.

[0023] The operation input unit 124 includes a controller and a touch panel (not shown) and receives user instruction inputs for the drone 100 and the display device 200. The user can operate the controller and the touch panel to instruct the drone 100 to turn, ascend, or descend, and to set the display settings for the display device 200.

[0024] The storage unit 125 temporarily stores image data received by the wireless communication unit 121 (output by the first imaging device 201 and the second imaging device 202) and display data generated by the display control unit 123. It also stores information obtained by the position acquisition unit 110, the velocity acquisition unit 111, the attitude acquisition unit 112, and the distance acquisition unit 113, and the display mode settings for the display unit 122 in the display control unit 123. It is also used as a storage area for programs executed by the control unit 120 (described later), a storage area for various parameters, and a work area during program execution.

[0025] The interface (I / F) unit 126 is an interface for connecting the display device 200 to an external device. Examples of the external device include a PC (Personal Computer), a storage medium (for example, a hard disk, a memory card, an SD card, a USB memory, etc.), and a display device such as a display.

[0026] The control unit 120 comprehensively controls the components of the display device 200 described above, and sets various parameters, controls display, and issues instructions for sending and receiving data.

[0027] FIG. 2 is a diagram showing the hardware configuration of the drone 100. The first imaging device 201 is an imaging device including a first imaging unit 102, a first image processing unit 103, and a first imaging control unit 104. Similarly, the second imaging device 202 is an imaging device including a second imaging unit 105, a second image processing unit 106, and a second imaging control unit 107. The motor driver 203 is a motor driver that constitutes the drive unit 109. The motor driver 203 sends drive signals to each motor attached to the drone 100 to enable navigation and attitude control of the drone 100. The wireless communication driver 204 is a driver for wireless communication that constitutes the wireless communication unit 108.

[0028] The GPS 205 is a GPS receiver that constitutes the position acquisition unit 110. The angular velocity sensor 206 is an angular velocity sensor that constitutes the attitude acquisition unit 112. The acceleration sensor 207 is an acceleration sensor that constitutes the velocity acquisition unit 111. The distance sensor 208 is a distance sensor that constitutes the distance acquisition unit 113. The electronic compass 209 is an electronic compass that constitutes the position acquisition unit 110.

[0029] The CPU 210 executes a control program to control various functions in addition to fulfilling the role of the control unit 101. The RAM 211 is a volatile memory and constitutes the storage unit 114. The ROM 212 is a non-volatile memory and constitutes the storage unit 114.

[0030] FIG. 3 is a diagram showing the hardware configuration of the display device 200. The display 301 is a display that constitutes the display unit 122. As will be described in detail later, the display 301 displays images acquired by each imaging device and information acquired by each information acquisition unit. The controller 302 is a controller that constitutes the operation input unit 124. The controller 302 controls the drone 100 and changes the display settings of the display device 200 (for example, by receiving instructions from a user). The I / F driver 303 constitutes the I / F unit 126, and when the display device 200 is connected to an external device via a wired connection, performs format conversion to match the standard of the device. The wireless communication driver 304 is a driver for wireless communication that constitutes the wireless communication unit 121.

[0031] The CPU 305 executes a control program to control various functions in addition to fulfilling the role of the control unit 120. The RAM 306 is a volatile memory and constitutes the storage unit 125. The ROM 307 is a non-volatile memory and constitutes the storage unit 125.

[0032] FIG. 4 is a diagram showing an example of the external appearance of the drone 100. The first imaging device 201 is an imaging device capable of capturing wide-angle images used for navigation control, and is attached to the drone 100 with a fixed angle of view. The second imaging device 202 is an imaging device capable of capturing images used for purposes other than navigation control. For example, the second imaging device 202 is an imaging device capable of zooming used to capture images of the surrounding environment, and the imaging angle of view can be changed depending on the distance and direction of the subject. The second imaging device 202 may be configured to have a drive mechanism for panning and tilting operations.

[0033] FIG. 5 is a diagram showing an example of the appearance of the display device 200. An example of a display on the display unit 122 (display 301) is also shown. The display unit 122 has a main display area 501 that displays an image captured by the first imaging device 201, which is a navigation image, and a sub-display area 502 that displays an image captured by the second imaging device 202, which is an image for obtaining surrounding information. Here, the navigation image refers to an image that a user (mainly an operator) references for the purpose of navigation control, and the surrounding information image refers to an image that a user references for purposes other than navigation control. While FIG. 5 shows an example in which the display area 502 is superimposed on the display area 501, a tiled arrangement (without superimposition) may also be used. Furthermore, while the display device 200 in FIG. 5 shows a configuration in which the controller 302 and the display 301 are integrated, the controller 302 and the display 301 may also be configured as separate devices. Furthermore, the operation input unit 124 may be realized by configuring the display 301 as a touch panel display.

[0034] <Device Operation> 6 and 7, a description will be given below of control of the display mode on the display unit 122 depending on the state of the drone 100. Figures 6 and 7 show an example of display control on the display unit 122.

[0035] Fig. 6 is a diagram illustrating display control in the display device 200 based on the movement speed of the drone 100. Fig. 6(a) shows the display when the drone 100 is traveling at a high speed (above a threshold value), and Fig. 6(b) shows the display when the drone 100 is traveling at a low speed (below the threshold value).

[0036] The speed information of the drone 100 is acquired by the speed acquisition unit 111, and the acquired speed information is transmitted from the drone 100 by the wireless communication unit 108 and received by the wireless communication unit 121 of the display device 200. The received speed information is input to the display control unit 123, and the display control unit 123 controls the display mode according to the speed information.

[0037] When the acquired speed is equal to or greater than a threshold, the image acquired from the drone 100 by the first imaging unit 102, which is the image for navigation, is displayed preferentially to improve operability for the operator. Therefore, as shown in FIG. 6(a), the sub display area is reduced as shown in display area 602. On the other hand, when the speed is less than the threshold, as shown in FIG. 6(b), the visibility of both the image acquired from the first imaging unit 102, which is the image for navigation, and the image acquired from the second imaging unit 105, which is the image for acquiring surrounding information, is ensured. Therefore, as shown in FIG. 6(b), the sub display area is expanded as shown in display area 502 (here, it is restored to the same size as in FIG. 5). In other words, using the size of the main display area as a reference, the relative size of the sub display area is set smaller in FIG. 6(a) than in FIG. 6(b).

[0038] In this way, when the speed of the drone 100 is high, a display that improves operability for the operator is provided, and when the speed of the drone 100 is low, a display that combines operability with visibility of the images from the two imaging devices is provided.

[0039] 7A and 7B are diagrams illustrating display control in the display device 200 based on the zoom magnification (imaging angle of view) of the second imaging unit 105. Fig. 7A shows a display when the second imaging unit 105 has a high magnification (zoom magnification equal to or greater than a threshold value), and Fig. 7B shows a display when the second imaging unit 105 has a low magnification (zoom magnification less than a threshold value).

[0040] The zoom magnification information of the second imaging unit 105 is acquired by the second imaging control unit 107, and the acquired zoom magnification information is transmitted from the drone 100 by the wireless communication unit 108 and received by the wireless communication unit 121 of the display device 200. The received zoom magnification information is input to the display control unit 123, and the display control unit 123 controls the display mode in accordance with the zoom magnification information.

[0041] When the zoom magnification obtained is equal to or greater than a threshold value (the imaging angle of view is equal to or smaller than a predetermined angle of view), the image of the second imaging device 202, which is the image for obtaining surrounding information obtained from the drone 100, is displayed preferentially to the operator. For this reason, as shown in FIG. 7(a), the sub display area is enlarged as shown in display area 702. On the other hand, when the zoom magnification is less than the threshold value (the imaging angle of view is larger than the predetermined angle of view), the image of the first imaging unit 102, which is the image for navigation, is displayed preferentially. For this reason, the sub display area is reduced as shown in display area 502 (here, it is returned to the same size as in FIG. 5). In other words, using the size of the main display area as a reference, the relative size of the sub display area is set larger in FIG. 7(a) compared to FIG. 7(b).

[0042] Fig. 8 is a flowchart of display control in the first embodiment. Specifically, it is a flowchart of control for changing the display mode on the display device 200 according to the speed of the drone 100 and the zoom magnification of the second imaging unit 105, as described with reference to Figs. 6 and 7.

[0043] In S1000, the control unit 120 acquires the velocity information of the drone 100 via the wireless communication unit 121. The velocity information is acquired by the velocity acquisition unit 111 and the attitude acquisition unit 112 of the drone 100 as described above.

[0044] In S1001, the control unit 120 determines the speed information acquired in S1000. If the speed of the drone 100 is equal to or greater than the threshold, the process proceeds to S1002. On the other hand, if the speed of the drone 100 is less than the threshold, the process proceeds to S1003.

[0045] In S1002, the display control unit 123 performs a process of changing the display mode. Here, since the speed of the drone 100 is equal to or greater than the threshold value, the display area 502 is reduced (to the display area 602 in FIG. 6(a)).

[0046] In S1003, the control unit 120 acquires zoom magnification information of the second imaging unit 105 via the wireless communication unit 121. The zoom magnification information is acquired by the second imaging control unit 107 of the drone 100 as described above. The zoom magnification is an optical zoom magnification by a zoom lens or a digital zoom magnification by cropping the acquired video. Note that the zoom magnification information may be acquired as information on the imaging angle of view.

[0047] In S1004, the control unit 120 determines the zoom magnification information acquired in S1003. If the zoom magnification of the second imaging unit 105 is equal to or greater than the threshold value, the process proceeds to S1005. On the other hand, if the zoom magnification of the second imaging unit 105 is less than the threshold value, the process proceeds to S1006.

[0048] In S1005, the display control unit 123 performs a process of changing the display mode. Here, since the zoom magnification of the second imaging unit 105 is equal to or greater than the threshold value, the display area 502 is enlarged (display area 702 in FIG. 7(a)).

[0049] In S1006, the display control unit 123 performs a process of changing the display mode. Here, since the speed of the drone 100 is less than the threshold value and the zoom magnification of the second imaging unit 105 is less than the threshold value, the display area 502 is displayed at the normal magnification (the display area 502 in FIGS. 6(b) and 7(b)).

[0050] 8 shows that after the determination based on the speed information of the drone 100, the determination based on the zoom magnification of the second imaging unit 105 is performed and the display mode is changed, but the determination order can be changed as desired. The determination order may be set in advance in the storage unit 125, or may be arbitrarily specified by the user by operating the operation input unit 124.

[0051] As described above, according to the first embodiment, the display mode on the display unit of multiple images acquired by a drone having multiple imaging devices is controlled according to the state of the drone (speed, zoom magnification). In particular, to improve the operability of the drone, when the speed of the drone is equal to or greater than a threshold, the image for acquiring surrounding information is displayed small, and the image for navigation is displayed relatively large. Furthermore, when the zoom magnification of the image for acquiring surrounding information is equal to or greater than a threshold, the image for acquiring surrounding information is displayed large, and the image for navigation is displayed relatively small.

[0052] (Variation) 6 and 7, the size of the sub display area in each state may be a size set in advance in the storage unit 125. Alternatively, the user may specify the size in each state by operating the operation input unit 124. In addition, the size of the sub display area may be changed in more stages depending on parameters such as the speed information of the drone 100 and the zoom magnification of the second imaging device 202.

[0053] 6 and 7, the size of the sub display area is enlarged and reduced, but the sub display area may be displayed or hidden by switching the display. If the sub display area and the main display area are arranged in a tiled configuration, the main display area may be displayed full screen when the sub display area is hidden. Furthermore, the sub display area and the main display area may be displayed on divided display unit 122.

[0054] In the first embodiment described above, the display device 200 controls the change of the display mode, but the control unit 101 of the drone 100 may directly instruct the display mode on the display unit 122 via the wireless communication unit 108.

[0055] Furthermore, in the first embodiment described above, the display mode is changed based on the speed information of the drone 100 and the zoom magnification of the second image capture device 202, but the determination of whether to change the display mode may be made based on other factors. For example, the distance between the drone 100 and a surrounding object acquired by the distance acquisition unit 113, the orientation of the second image capture unit 105, etc. may be used for the determination. Furthermore, if the second image capture unit 105 has a drive mechanism and is configured to be able to change the shooting direction (such as by panning or tilting), the operating state of the drive mechanism (operating or not operating) or the setting state (enabled or disabled) of the automatic tracking function or object recognition function may be used for the determination.

[0056] (Second embodiment) In the first embodiment, a configuration was described in which two images received from a drone are displayed in two display areas on the display unit, and the display size is changed. In the second embodiment, a case will be described in which the number of images (and information) that can be displayed on the display unit 122 is greater than the number of display areas. In addition to the two images in the first embodiment, the images and information that can be displayed include invisible light images, images of information about the surroundings of the drone 100 acquired by the distance acquisition unit 113, and images of position information of the drone 100 acquired by the position acquisition unit 110. Examples of invisible light images include near-infrared images and far-infrared (thermal) images.

[0057] <System configuration and device configuration> The system configuration and the configuration of each device (drone, display device) are almost the same as those in the first embodiment. However, the difference is that the second image capturing device 202 of the drone has two units: a visible light capturing unit that captures visible light video and an infrared light capturing unit that captures infrared light video. The visible light capturing unit and the infrared light capturing unit are configured to capture substantially the same capturing area (capturing angle of view). Another difference is that the control unit 101 of the drone transmits various pieces of information (here, surrounding information and position information) acquired by the drone to the display device as information to be displayed on the display unit of the display device.

[0058] <Device Operation> Fig. 9 is a diagram illustrating display control in a display device. Fig. 9 shows an example of a display on display unit 122. Display unit 122 shows a main display area 901 that displays navigation images, and sub-display areas (display area 902 and display area 903) that display other images and information. Here, display area 902 and display area 903 are superimposed on display area 901.

[0059] The main display area 901 displays an image (navigation image) acquired by the first imaging device 201. On the other hand, the sub display area may display a visible light image acquired by the visible light imaging unit, an infrared light image acquired by the infrared light imaging unit, surrounding information of the drone 100 acquired by the distance acquisition unit 113, and position information of the drone 100 acquired by the position acquisition unit 110. That is, here, the number of sub display areas (two) is less than the total number (four) of captured images and status information that can be displayed in the sub display areas.

[0060] For each video and information, a "display order" and a "change condition for changing the display order" are set, which are given default settings for the sub display area. If there is no video and information that satisfies the change condition, the display content to be displayed in the sub display area is determined (assigned) based on the display order of the default settings. If there is video and information that satisfies the change condition, the priority of the video or information that satisfies the change condition is increased (the display order is controlled), and the rest are displayed in the sub display area according to the display order of the default settings. Here, it is assumed that the video / information with the first display order is displayed in display area 902, and the video / information with the second display order is displayed in display area 903.

[0061] In this embodiment, the default display order is set as follows: location information (1st), visible light image (2nd), surrounding information (3rd), and infrared image (4th). Furthermore, as change conditions, "surrounding objects and the drone are close to each other within a specified distance" is set for surrounding information, and "visibility of visible light image is poor" is set for infrared image. Visibility of visible light image is determined, for example, by referring to the brightness and histogram of the visible light image acquired by the visible light imaging unit of the second imaging device 202. Then, if the change conditions are met, the priority is raised by two levels (the order is raised by two).

[0062] 9(a) shows a display state when there are no objects nearby the drone and the visibility of the image captured by the visible light imaging unit of the second imaging device 202 is good. Therefore, according to the default display order, "location information" is displayed in display area 902, and "visible light image" is displayed in display area 903.

[0063] FIG. 9(b) shows the display state when the state shown in FIG. 9(a) changes to a state where the surrounding object and the drone are close enough to each other to be within a predetermined distance. Because the change condition for the "surrounding information" is met, the priority of the "surrounding information" increases and its display order is changed to first place. As a result, the "surrounding information" is displayed in display area 902, and the "location information" is displayed in display area 903.

[0064] FIG. 10 is a flowchart of display control in the second embodiment.

[0065] In S2000, the control unit 120 acquires the surrounding information of the drone 100 acquired by the distance acquisition unit 113 via the wireless communication unit 121.

[0066] In S2001, the control unit 120 determines the distance between the drone 100 and the surrounding object based on the surrounding information of the drone 100 acquired in S2000. If the distance between the drone 100 and the surrounding object is equal to or less than a threshold, the process proceeds to S2002. If the distance between the drone 100 and the surrounding object is greater than the threshold, the process proceeds to S2004.

[0067] In S2002, the display control unit 123 performs processing to change the display mode of the display area 902. Here, the change condition for the "surrounding information" is satisfied, so the "surrounding information" is displayed in the display area 902 (display area 902 in FIG. 9(b)). Note that the surrounding information here is, for example, a visible light image onto which distance information to objects detected in the surroundings is superimposed. The display area 902 in FIG. 9(b) shows a state in which two objects, an object aa meters (m) away and an object bb meters (m) away, have been detected in the surroundings. This allows the operator to recognize that there are surrounding objects near the drone 100, and also allows the operator to easily grasp the surrounding information of the drone 100.

[0068] In S2003, the display control unit 123 performs processing to change the display mode of the display area 903. Here, "location information," which has the next highest display order after "surrounding information" (the default display order is first), is displayed in the display area 903 (display area 903 in FIG. 9(b)).

[0069] In S2004, the control unit 120 acquires (derives) the brightness of the visible light image captured by the visible light imaging unit of the second imaging device 202.

[0070] In S2005, the control unit 120 determines the visibility of the visible light image based on the brightness of the visible light image acquired in S2004. For example, if the brightness of the visible light image is equal to or greater than a threshold, the control unit 120 determines that the visibility is good and proceeds to S2006. On the other hand, if the brightness of the visible light image is less than the threshold, the control unit 120 determines that the visibility is poor (the image is too dark) and proceeds to S2008.

[0071] In S2006, the display control unit 123 performs processing to change the display mode of the display area 902. Here, since neither the change conditions for "ambient information" nor "infrared light image" are met, "location information" is displayed in the display area 902 according to the default display order (first place) (display area 902 in FIG. 9(a)).

[0072] In S2007, the display control unit 123 performs processing to change the display mode of the display area 903. Here, since neither the change conditions for "ambient information" nor "infrared light image" are met, the "visible light image" is displayed in the display area 903 according to the default display order (second place) (display area 903 in FIG. 9(a)).

[0073] In S2008, the display control unit 123 performs processing to change the display mode of the display area 902. Here, the change condition for "ambient information" is not met, but the change condition for "infrared light image" is met. However, even in this case, the display order of "infrared image" after the change is "second place." Therefore, "location information" is displayed in the display area 902.

[0074] In S2009, the display control unit 123 performs processing to change the display mode of the display area 903. Here, as described above, the display order of the "infrared image" after the change is now "second place." Therefore, the "infrared image" is displayed in the display area 903. This allows the operator to capture the "infrared image" instead, even if the "visible light image" is dark and has poor visibility.

[0075] As described above, according to the second embodiment, when the number of display areas is smaller than the number of images / information that can be displayed in the display areas, the display mode on the display unit is controlled according to the environment (surrounding environment, captured images). For example, to improve drone operability, when the state changes to "the surrounding objects and the drone are close to each other within a predetermined distance," priority is given to displaying surrounding information. Also, when the state changes to "poor visibility of visible light images," priority is given to displaying infrared light images. This makes it possible to provide the operator with the necessary information while preventing a decrease in visibility / operability of the display unit, which would be a problem if all images / information that can be displayed were displayed.

[0076] (Variation) 9 has been described as a form in which the video / information displayed in the sub display areas (display area 902 and display area 903) is changed, but the display may also be changed including the main display area (display area 901). Also, while an example in which two sub display areas are superimposed on one main display area is shown in FIG. 9, each of the sub display areas may be divided into display unit 122 according to the number of display areas. Also, the user may arbitrarily set the area in which the display is fixed and the area in which the display is changed.

[0077] 9 shows an example in which "location information" is changed from display area 902 to display area 903, but the display area may be controlled not to be changed for the video and information (here, "location information") that are continuously displayed before and after the determination. Also, the control of changing the video / information to be displayed in the second embodiment may be combined with the control of changing the display size in the first embodiment.

[0078] Furthermore, the "threshold value for the distance between the drone 100 and the surrounding object" and the "threshold value for the brightness of the visible light image" may be set in advance in the storage unit 125. Furthermore, the "threshold value for the distance between the drone 100 and the surrounding object" may be variable depending on the speed of the drone 100. In determining the visibility of the visible light image, the determination is made based on the brightness from the luminance information, but a luminance histogram of the visible light image may also be calculated and used to determine the visibility.

[0079] 9 and 10, the default display order of each video / information, the video / information to be displayed, and the order of determining whether to change the display mode can be changed arbitrarily. The change conditions may be set in advance in storage unit 125, or may be arbitrarily specified by the user by operating operation input unit 124.

[0080] The disclosure of this specification includes the following control device, control method, and program. (Item 1) an image acquisition means for acquiring a first captured image acquired by a first imaging unit mounted on the drone and used for navigation control of the drone, and a second captured image acquired by a second imaging unit mounted on the drone and used for purposes other than the navigation control; A status acquisition means for acquiring a status of the drone; a control means for controlling a display mode when the first captured image and the second captured image are displayed on a display unit based on the state obtained by the state obtaining means; A display control device comprising: (Item 2) the control means is configured to display the first captured image in a first display area of ​​the display unit, and to display the second captured image in a second display area of ​​the display unit; The control means performs control to change the size of at least one of the first display area and the second display area depending on the change in the state, or performs control to switch between displaying and hiding the second captured image in the second display area. 2. The display control device according to item 1, (Item 3) The state includes a movement speed of the drone; The control means controls the size of at least one of the first display area and the second display area so that the relative size of the second display area based on the size of the first display area is smaller when the moving speed is equal to or greater than a predetermined threshold value than when the moving speed is less than the predetermined threshold value. 3. The display control device according to item 2, (Item 4) the state includes an imaging angle of the second imaging unit, The control means controls the size of at least one of the first display area and the second display area so that the relative size of the second display area based on the size of the first display area is larger when the imaging angle of view is equal to or smaller than a predetermined angle of view than when the imaging angle of view is larger than the predetermined angle of view. 4. The display control device according to item 2 or 3. (Item 5) the second imaging unit has a drive mechanism for changing the imaging direction, the state includes an operating state of the drive mechanism or a setting state of an automatic tracking function via the drive mechanism, The control means controls the size of at least one of the first display area and the second display area so that the relative size of the second display area based on the size of the first display area is larger when the drive mechanism is operating or an automatic tracking function via the drive mechanism is enabled than when the drive mechanism is not operating or an automatic tracking function via the drive mechanism is disabled. 4. The display control device according to item 2 or 3. (Item 6) the size of the second display area is smaller than the size of the first display area; The second display area is an area that overlaps with the first display area. 6. A display control device according to any one of items 2 to 5, characterized in that: (Item 7) the second captured image is a visible light image, The image acquisition means further acquires a third captured image, which is an infrared image acquired by a third imaging unit mounted on the drone and captures the same imaging area as the second captured image, The control means further controls a display mode when the third captured image is displayed on the display unit based on the state obtained by the state obtaining means. 2. The display control device according to item 1, (Item 8) the control means further displays status information regarding the status acquired by the status acquisition means on the display unit; the control means displays the first captured image in a first display area of ​​the display unit, and determines, based on a given default setting, which of the second captured image, the third captured image, and the status information display content to be assigned to each of a second display area and a third display area of ​​the display unit; The control means changes the display content of at least one of the second display area and the third display area depending on the change in the state. 8. A display control device according to item 7, characterized in that: (Item 9) The state acquisition means acquires a position of the drone and a distance between the drone and a surrounding object, When the distance is equal to or less than a predetermined threshold, the control means displays the position of the drone in either the second display area or the third display area. 9. A display control device according to item 8. (Item 10) the status acquisition means acquires information regarding visibility of the second captured image, When the visibility is poor, the control means displays the third captured image in either the second display area or the third display area. 9. A display control device according to item 8. (Item 11) A control method for a display control device that controls display modes of a plurality of images on a display unit, comprising: an image acquisition process for acquiring a first image acquired by a first imaging unit mounted on the drone and used for navigation control of the drone, and a second image acquired by a second imaging unit mounted on the drone and used for purposes other than the navigation control; A status acquisition step of acquiring a status of the drone; a control step of controlling a display mode when the first captured image and the second captured image are displayed on the display unit based on the state obtained in the state obtaining step; A control method comprising: (Item 12) Item 12. A program for causing a computer to execute the control method according to Item 11.

[0081] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0082] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0083] 10 System; 100 Drone; 200 Display device; 120 Control unit; 121 Wireless communication unit; 122 Display unit; 123 Display control unit; 124 Operation input unit; 125 Memory unit; 126 I / F unit

Claims

1. an image acquisition means for acquiring a first captured image acquired by a first imaging unit mounted on the drone and used for navigation control of the drone, and a second captured image acquired by a second imaging unit mounted on the drone and used for purposes other than the navigation control; A status acquisition means for acquiring a status of the drone; a control means for controlling a display mode when the first captured image and the second captured image are displayed on a display unit based on the state obtained by the state obtaining means; A display control device comprising:

2. the control means is configured to display the first captured image in a first display area of ​​the display unit and to display the second captured image in a second display area of ​​the display unit; The control means performs control to change the size of at least one of the first display area and the second display area depending on the change in the state, or performs control to switch between displaying and hiding the second captured image in the second display area.

2. The display control device according to claim 1.

3. The state includes a movement speed of the drone; The control means controls the size of at least one of the first display area and the second display area so that the relative size of the second display area based on the size of the first display area is smaller when the moving speed is equal to or greater than a predetermined threshold value than when the moving speed is less than the predetermined threshold value.

3. The display control device according to claim 2.

4. the state includes an imaging angle of the second imaging unit, The control means controls the size of at least one of the first display area and the second display area so that the relative size of the second display area based on the size of the first display area is larger when the imaging angle of view is equal to or smaller than a predetermined angle of view than when the imaging angle of view is larger than the predetermined angle of view.

3. The display control device according to claim 2.

5. the second imaging unit has a drive mechanism for changing the imaging direction, the state includes an operating state of the drive mechanism or a setting state of an automatic tracking function via the drive mechanism, The control means controls the size of at least one of the first display area and the second display area so that the relative size of the second display area based on the size of the first display area is larger when the drive mechanism is operating or an automatic tracking function via the drive mechanism is enabled than when the drive mechanism is not operating or an automatic tracking function via the drive mechanism is disabled.

3. The display control device according to claim 2.

6. the size of the second display area is smaller than the size of the first display area; The second display area is an area that overlaps with the first display area.

3. The display control device according to claim 2.

7. the second captured image is a visible light image, The image acquisition means further acquires a third captured image, which is an infrared image acquired by a third imaging unit mounted on the drone and captures the same imaging area as the second captured image, The control means further controls a display mode when the third captured image is displayed on the display unit based on the state obtained by the state acquisition means.

2. The display control device according to claim 1.

8. the control means further displays status information regarding the status acquired by the status acquisition means on the display unit; the control means displays the first captured image in a first display area of ​​the display unit, and determines, based on a given default setting, which display content of the second captured image, the third captured image, or the status information to assign to and display in each of the second display area and the third display area of ​​the display unit; The control means changes the display content of at least one of the second display area and the third display area depending on the change in the state.

8. The display control device according to claim 7,

9. The state acquisition means acquires a position of the drone and a distance between the drone and a surrounding object, When the distance is equal to or less than a predetermined threshold, the control means displays the position of the drone in either the second display area or the third display area.

9. The display control device according to claim 8.

10. the state acquisition means acquires information regarding visibility of the second captured image, When the visibility is poor, the control means displays the third captured image in either the second display area or the third display area.

9. The display control device according to claim 8.

11. A control method for a display control device that controls display modes of a plurality of images on a display unit, comprising: an image acquisition process for acquiring a first image acquired by a first imaging unit mounted on the drone and used for navigation control of the drone, and a second image acquired by a second imaging unit mounted on the drone and used for purposes other than the navigation control; A status acquisition step of acquiring a status of the drone; a control step of controlling a display mode when the first captured image and the second captured image are displayed on the display unit based on the state obtained in the state obtaining step; A control method comprising:

12. A program for causing a computer to execute the control method according to claim 11.

Images