Imaging system, imaging method, and mobile body equipped with the imaging system

Abstract

The imaging system mounted on the drone reduces the chances of losing sight of the subject being automatically tracked during automatic tracking, thereby ensuring reliable capture of the subject. [Solution] An imaging system for capturing images used for automatic tracking of a subject by a moving object, comprising: a visible light imaging unit that captures visible light images and a non-visible light imaging unit that captures non-visible light images whose shooting area overlaps with that of the visible light imaging unit in at least a portion thereof; a score is calculated for the visible light image captured by the visible light imaging unit and the non-visible light image captured by the non-visible light imaging unit for the same subject; and based on the result of comparing the calculated scores, either the visible light imaging unit or the non-visible light imaging unit is selected as the imaging unit to be used for controlling the automatic tracking of the subject by the moving object.

Description

Technical Field

[0001] The present invention relates to an imaging system, and particularly to an imaging system suitable for reliably capturing a subject when mounted on a drone and performing automatic tracking of the subject.

Background Art

[0002] A drone is an aircraft (unmanned aerial vehicle) that can move in response to a wireless operation or autonomously. Technologies related to drones have developed rapidly, and their applications cover a wide range, including commercial, civilian, and military uses, and are expected to be utilized in many fields.

[0003] Particularly, it has become common to mount an imaging device (camera) on a drone to photograph a subject (person, animal, vehicle, etc.) located far away. At this time, as a technology for changing the imaging area of the imaging device based on the position and movement of the subject and performing tracking, a technology called automatic tracking is known.

[0004] On the other hand, in order for the camera mounted on the drone to acquire an image even in a poor visibility environment such as fog or haze, in addition to a visible light camera, a non-visible light camera (thermal camera, etc.) may be mounted. Also, since the information that can be obtained by a visible light camera and a non-visible light camera has different wavelengths of electromagnetic waves used for imaging, depending on the environment, background, and subject, the results of image analysis of the two may be different.

[0005] As a technology for using visible light images and non-visible light (infrared) images for monitoring purposes, for example, there is a disclosure in Patent Document 1. In the imaging device described in Patent Document 1, depending on the mode of object detection, one of a visible light image, a non-visible light image, or a composite image thereof is determined as an output image for monitoring.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] The technology described in Patent Document 1 outputs the selected appropriate image to the client device, thereby enabling the user to obtain an image that is easy to view.

[0008] However, in automatic tracking, the imaging device analyzes the image and controls the drive unit, so it cannot output a suitable image during the automatic tracking stage. As a result, there was a risk of losing track of the subject because proper image analysis could not be performed.

[0009] The objective of the present invention is to provide an imaging system mounted on a drone that can reliably capture a subject by reducing the chance of losing sight of the subject being automatically tracked when performing automatic tracking of the subject. [Means for solving the problem]

[0010] Preferably, the imaging system of the present invention is an imaging system for capturing images used for automatic tracking of a subject by a moving object, and the moving object comprises a visible light imaging unit that captures visible light images and a non-visible light imaging unit that captures non-visible light images whose shooting area overlaps with that of the visible light imaging unit in at least a portion of the area. A score is calculated for the visible light image captured by the visible light imaging unit and the non-visible light image captured by the non-visible light imaging unit for the same subject, and based on the result of comparing the calculated scores, either the visible light imaging unit or the non-visible light imaging unit is selected and selected as the imaging unit to be used for controlling the automatic tracking of the subject by the moving object. [Effects of the Invention]

[0011] According to the present invention, when an imaging system mounted on a drone performs automatic tracking of a subject, it is possible to provide an imaging system that can reduce the chance of losing sight of the subject being automatically tracked and can reliably capture the subject. [Brief explanation of the drawing]

[0012] [Figure 1] This is an overall configuration diagram of a drone system according to one embodiment. [Figure 2] This is a hardware configuration diagram of the imaging system. [Figure 3] This is a hardware configuration diagram of the client device. [Figure 4] This is a perspective view of the drone's exterior. [Figure 5A] This figure shows how a visible light image is displayed in the user interface of a client device. [Figure 5B] This figure shows how a non-visible light image is displayed in the user interface of a client device. [Figure 6] This diagram illustrates how a drone automatically tracks a subject. [Figure 7A] This is a diagram (part 1) showing an example of how images captured during automatic tracking are displayed. [Figure 7B] This is a diagram (part two) showing an example of how images captured during automatic tracking are displayed. [Figure 8] This is a flowchart showing the process by which a drone automatically tracks a subject. [Figure 9A] This figure shows an example of displaying visible light and invisible light images in the display area of ​​a client device. [Figure 9B] This figure shows an example of a user interface for selecting images to display. [Figure 10A] This figure shows an example of a visible light image when the subject is an animal. [Figure 10B] This figure shows an example of a visible light image when the subject is a vehicle. [Figure 11] This diagram shows a user interface in which the user selects a subject to be automatically tracked on the client device. [Figure 12A] This figure shows an example of displaying the edge detection score for a subject in a visible light image. [Figure 12B] It is a diagram showing an example of displaying a score for edge detection of a subject in a non-visible light image.

Embodiment for Carrying out the Invention

[0013] Hereinafter, an embodiment according to the present invention will be described with reference to FIGS. 1 to 12B. First, the configuration of the drone system according to an embodiment will be described with reference to FIGS. 1 to 3. FIG. 1 is an overall configuration diagram of a drone system according to an embodiment. FIG. 2 is a hardware configuration diagram of the imaging system. FIG. 3 is a hardware configuration diagram of the client device.

[0014] As shown in FIG. 1, the drone system 1 has a configuration in which the drone 10 and the client device 200 are connected by the network 20.

[0015] The network 20 usually communicates wirelessly, and for example, communication standards such as Wi-Fi (2.4 GHz / 5 GHz), LTE (4G / 5G), and RF communication are used.

[0016] The drone 10 is a so-called unmanned aerial vehicle and is used for various purposes as described above. In this embodiment, the drone is assumed to have an imaging function and functions for shooting and tracking the movement of a subject.

[0017] The client device 200 is a device for a user (operator) to operate the drone 10 and display information collected by the drone 10. The client device 200 may be a dedicated operation device for the drone or may be realized by a general personal computer.

[0018] The drone 10 consists of an imaging system 100 and a navigation system 500. The imaging system 100 is a system that collects information using its mounted camera and takes images of the surroundings or specific subjects while the drone is flying. The navigation system 500 is a system for navigating the drone 10. In the example shown in Figure 1, the control unit 101, communication unit 102, and memory unit 103 are shared and used to perform their respective functions in both systems.

[0019] The control unit 101 is a functional unit that controls each part of the imaging system 100 and performs image processing, as well as controlling each part of the navigation system 500 and performing processing related to the navigation of the drone 10.

[0020] The communication unit 102 is a functional unit that communicates between the imaging system 100 and navigation system 500 of the drone 10 and the client device 200. The communication unit 102 can transmit image data output by the first image processing unit 113 and the second image processing unit 123 (described later) to the client device 200 via the network 20. It can also receive operation information input from the operation input unit 205 of the client device 200. Furthermore, it can receive navigation-related commands from the user and exchange navigation-related information via the network 20.

[0021] The memory unit 103 is a functional unit that stores necessary data and programs for the drone 10's imaging system 100 and navigation system 500. The memory unit 103 can store and read image data output by the first image processing unit 113 and the second image processing unit 123. Furthermore, it is also used as a storage area for programs executed by the control unit 101, a storage area for various parameters, and a work area during program execution.

[0022] The imaging system 100 consists of a control unit 101, a first imaging unit 110, a second imaging unit 120, a communication unit 102, a storage unit 103, an imaging drive unit 140, a score calculation unit 151, and a movement vector calculation unit 152.

[0023] The first imaging unit 110 and the second imaging unit 120 are camera units with different roles, each installed on the drone, and are functional units that perform imaging. Specific hardware examples on the drone 10 will be described later.

[0024] The first imaging unit 110 has a first visible light imaging unit 111 and a first non-visible light imaging unit 112. These units each image electromagnetic waves at different wavelengths; the first visible light imaging unit 111 is in the visible light wavelength range (approximately 360 nm to 830 nm), while the first non-visible light imaging unit 112 is, for example, in the infrared wavelength range (approximately 830 nm to 15000 nm).

[0025] The first image processing unit 113 is a functional unit that converts the signals photoelectrically converted by the first visible light imaging unit 111 and the first non-visible light imaging unit 112 into image data (digital data).

[0026] The functional configuration of the second imaging unit 120 is the same as that of the first imaging unit 110, and each of its functions is also the same.

[0027] In the first image processing unit 113 and the second image processing unit 123, pixel data is converted into a digital signal by A / D conversion. The converted digital signal is converted back into image data through correction and development processes such as black level correction, gamma curve adjustment, temperature correction, scratch correction, noise reduction, and white balance correction. Data compression processes such as MP4 and JPEG are also performed. Furthermore, these image processing units perform corrections tailored to the wavelength characteristics of each image sensor.

[0028] The imaging drive unit 140 is a functional unit that drives the pan (horizontal movement of the camera head) and tilt (vertical movement of the camera head) of the camera unit mounted on the drone. It also drives the lens, aperture mechanism, and zoom of the imaging unit.

[0029] The score calculation unit 151 is a functional unit that calculates a score indicating whether it is preferable to use the visible light image from the visible light imaging unit or the invisible light image from the invisible light imaging unit as the image for tracking. The process for calculating the score at the edges of the visible light image and the invisible light image will be explained in detail later.

[0030] The motion vector calculation unit 152 is a functional unit that calculates motion vectors from image data. The process for calculating motion vectors will be explained in detail later.

[0031] The navigation system 500 consists of a control unit 101, a communication unit 102, a memory unit 103, a mobile drive unit 501, a position acquisition unit 502, a speed acquisition unit 503, an attitude acquisition unit 504, and a distance acquisition unit 505.

[0032] The mobile drive unit 501 is a functional unit that drives the drone 10 using motors, propellers, etc., to enable the drone 10 to navigate and control its attitude. The drone 10 can normally move forward and backward, left and right, and up and down while in flight. The mobile drive unit 501 receives operation information from the operation input unit 205 (described later) of the client device 200, as well as information obtained from the position acquisition unit 502, speed acquisition unit 503, attitude acquisition unit 504, and distance acquisition unit 505, and is controlled by the control unit 101 based on this information.

[0033] The position acquisition unit 502 is a functional unit that receives and processes GPS (Global Positioning System) signals to determine the position of the drone 10, and processes electronic compass signals to determine the direction.

[0034] The speed acquisition unit 503 is a functional unit that processes digital data output from an acceleration sensor and other sources to calculate the acceleration and velocity acting on the drone 10.

[0035] The attitude acquisition unit 504 is a functional unit that processes digital data output from an angular velocity sensor and the like to calculate the rotation and orientation changes of the drone 10.

[0036] The distance acquisition unit 505 is a functional unit that processes data output from distance sensors such as Lidar (Light Detection and Ranging), millimeter-wave radar, and ultrasonic sensors to calculate the surrounding environment of the drone 10 and the distance to obstacles. Alternatively, the distance acquisition unit 505 may calculate the distance to the target from the focus evaluation value obtained by phase-difference AF (AUTO FOCUS).

[0037] The client device 200 consists of a control unit 201, a communication unit 202, a display unit 203, a display control unit 204, an operation input unit 205, a storage unit 206, and an external device I / F unit 207.

[0038] The control unit 201 is a functional unit that controls each component of the client device 200 and performs system control such as setting various parameters, display control, and data transmission / reception instructions.

[0039] The communication unit 202 is a functional unit that communicates with the drone 10 via the network 20.

[0040] The display unit 203 is controlled by the display control unit 204 and is a functional unit that displays images captured by each imaging device and various information, and is implemented using an LCD (Liquid Crystal Display).

[0041] The display control unit 204 is a functional unit that controls the display of the display unit 203. The display control unit 204 selects the video data and information to be displayed on the display unit 203, controls the display format such as superimposed display and split display, and controls the display area of ​​each video and piece of information. It may also change the video and information displayed on the display unit 203, the display format, and the display area based on the operator's instructions. Furthermore, the display control unit 204 can switch the display format of the image displayed on the display unit 203 (visible light image and invisible light image).

[0042] The operation input unit 205 is a functional unit that inputs user instructions to the imaging system 100 and the client device 200, and is implemented by a controller (described later) or a touch panel. The user can operate these controllers or touch panels to control the drone's imaging system 100, navigation system 500, and client device 200.

[0043] The storage unit 206 is a functional unit that stores the program executed by the client device 200, storage areas for various parameters, and work data during program execution. The storage unit 206 can temporarily store and read image data output by the first imaging unit 110 and the second imaging unit 120, which are received by the communication unit 202, and display data generated by the display control unit 204.

[0044] The external device interface (I / F) section 207 is an interface for connecting the client device 200 to a display device such as a PC (Personal Computer) or a display. The external device interface (I / F) section 207 is also an interface for connecting to an external storage medium (e.g., a hard disk, memory card, SD card, USB memory, etc.).

[0045] In this embodiment, an example in which the imaging system 100 is mounted on a drone 10 has been described, but the embodiment is not limited to this, and may also be mounted on a mobile body such as an aircraft, ship, or vehicle.

[0046] Next, we will explain the hardware configuration of the imaging system using Figure 2. As shown in Figure 2, the imaging system 100 is configured such that a first imaging device 310, a second imaging device 320, an imaging drive mechanism 330, a CPU (Central Processing Unit) 301, a main memory 302, a non-volatile memory 303, and a communication I / F device 340 are connected by a bus.

[0047] The first imaging device 310 comprises a first visible light imaging mechanism 311a and a first invisible light imaging mechanism 311b. The first visible light imaging mechanism 311a comprises a visible light imaging optical system 312a and a visible light image sensor 313a. Similarly, the first invisible light imaging mechanism 311b comprises an invisible light imaging optical system 312b and an invisible light image sensor 313b.

[0048] The visible light imaging optical system 312a includes a visible light lens 314a (zoom lens, focus lens) and an aperture mechanism, and focuses visible light (wavelength: approximately 360nm to 830nm) from the subject onto the light-receiving surface of the visible light image sensor 313a. Here, the visible light lens is a lens with high transmittance in the visible light wavelength band, and the visible light image sensor 313a is an element with high sensitivity in the visible light wavelength band.

[0049] Similarly, the invisible light imaging optical system 312b includes an invisible light lens 314b (zoom lens, focus lens) and an aperture mechanism, which focuses invisible light (wavelengths other than visible light, for example, infrared wavelengths of approximately 830nm to 15000nm) from the subject onto the light-receiving surface of the invisible light image sensor 313b. Here, the invisible light lens is a lens with high transmittance in the invisible light wavelength band, and the invisible light image sensor 313b is an element with high sensitivity in the invisible light wavelength band.

[0050] Here, a zoom lens is a lens that moves along the optical axis and can change the magnification of the image, and a focus lens is a lens that moves along the optical axis and can adjust the focus. The aperture mechanism is a mechanism that adjusts the amount of light passing through the optical system.

[0051] The visible light image sensor 313a and the non-visible light image sensor 313b are semiconductor devices such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors. The visible light image sensor 313a and the non-visible light image sensor 313b convert the light incident from their respective imaging optics into an analog video signal by photoelectric conversion. In particular, the non-visible light image sensor 313b is an infrared sensor that is sensitive to infrared light, including near-infrared, mid-infrared, and far-infrared. Depending on the application, it may also be a semiconductor device that is sensitive to wavelengths other than visible light, such as an ultraviolet sensor.

[0052] The second imaging device 320, like the first imaging device 310, also includes a second visible light imaging mechanism 321a and a second non-visible light imaging mechanism 321b.

[0053] The functions and configurations of the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b are the same as those of the first visible light imaging mechanism 311a and the first invisible light imaging device described above.

[0054] The CPU 301 is a processor that controls and processes each function of the imaging system 100. The CPU 301 controls each function, such as the control unit 101, the first image processing unit 113, the second image processing unit 123, and the drive control of the imaging drive unit 140, as shown in the functional diagram of Figure 1.

[0055] Main memory 302 is a volatile semiconductor element such as RAM (Random Access Memory) and stores programs and work data executed by the CPU 301. Non-volatile memory 303 is a non-volatile semiconductor element such as flash memory and stores setting data and installed programs for the imaging system 100.

[0056] The imaging drive mechanism 330 consists of a motor 331, a gear 332, a pan-tilt mechanism 333, a motor driver 334, and an encoder 335.

[0057] The motor driver 334 is a driver that controls the drive of the motor 331. The motor 331 is a rotational mechanism controlled by the motor driver 334 that drives the pan-tilt mechanism 333. The gear 332 is a transmission mechanism that transmits the rotation of the motor to the movable part (pan-tilt mechanism 333). The encoder 335 is an element for detecting the rotational position of the pan-tilt mechanism 333, and the position of the pan-tilt mechanism 333 is detected by an optical encoder or a magnetic encoder. Alternatively, to realize the function of the encoder, a reference position may be calculated using a photo interrupter or the like, and the relative position of the pan-tilt mechanism 333 may be calculated from the control amount of the motor driver 334 based on the reference position. The pan-tilt mechanism 333 is a mechanism for performing pan and tilt movements in the camera head.

[0058] The communication I / F device 340 is an interface device for the imaging system 100 to communicate with other devices such as the client device 200.

[0059] In this embodiment, the imaging system 100 has a hardware configuration in which the CPU 301 executes a program installed in the non-volatile memory 303. However, it is not limited to this, and can also be implemented using a single logic circuit element (for example, an ASIC (Application Specific Integrated Circuit)).

[0060] Next, we will explain the hardware configuration of the client device using Figure 3. As shown in Figure 3, the client device 200 is configured such that a CPU 401, main memory 402, non-volatile memory 403, display I / F device 404, external device I / F device 405, communication I / F device 406, and input / output I / F device 407 are connected by a bus.

[0061] The CPU 401 controls the various parts of the client device 200 and executes programs. The main memory 402 is a volatile semiconductor element that stores programs executed by the CPU and work data. The non-volatile memory 403 is a non-volatile semiconductor element such as flash memory that stores programs executed by the client device and client device configuration data. The display I / F device 404 is an interface device for connecting a display device 410 such as a display. The external device I / F device 405 is an interface device for connecting the client device and an external device, and performs format conversion according to the standard when the client device 200 is connected to an external device via a wired connection. The communication I / F device 406 is a device for wirelessly connecting the client device 200 and the drone 10. The input / output I / F device 407 is an interface device for connecting input / output devices such as the controller 420.

[0062] Next, we will describe the appearance of the drone using Figure 4. Figure 4 is a perspective view of the drone's exterior.

[0063] This description of the embodiment will focus on the appearance of the drone 10 and the systems mainly related to the imaging system 100. As shown in Figure 4, the drone 10 is equipped with a navigation camera 150 and a payload camera 160 as imaging devices.

[0064] The navigation camera 150 is a camera mounted on the drone 10 to assist in its flight and support piloting and autonomous flight. It is a wide-angle camera mounted on the front of the drone 10. In other words, it is a fixed-angle camera capable of capturing wide-angle images used for navigation.

[0065] The payload camera 160 is a camera mounted on the drone 10 for taking photographs and observations. It is a camera specialized for photography and data collection, mounted on the underside of the drone 10, and is capable of changing the zoom magnification and shooting direction. In other words, it is a telephoto camera used to photograph the surrounding environment, and its zoom mechanism allows the zoom magnification (angle of view) to be changed depending on the distance and direction of the subject being photographed. Furthermore, the payload camera 160 has a drive mechanism and is configured to perform panning and tilting movements (changing the shooting direction).

[0066] Furthermore, the navigation camera 150 and the payload camera 160 may be mounted in positions other than those shown in Figure 4, as long as they can perform their respective functions.

[0067] Here, we will explain the relationship between the navigation camera 150 and payload camera 160 shown in Figure 4 and the functional configuration shown in Figure 1. In this embodiment, the first imaging unit 110 shown in Figure 4 corresponds to the payload camera 160, and the payload camera 160 corresponds to the second imaging unit 120 shown in Figure 4. As shown in Figure 4, the first imaging unit 110 has a first visible light imaging unit 111 and a first invisible light imaging unit 112, and the second imaging unit 120 has a second visible light imaging unit 121 and a second invisible light imaging unit 122. Therefore, it is assumed that both the navigation camera 150 and the payload camera 160 have the function of imaging in visible light and invisible light (for example, imaging in infrared).

[0068] As an example of using invisible light, a typical infrared camera is a camera that detects infrared radiation (thermal radiation) emitted by an object and converts it into an image. Another well-known example of an infrared camera is a thermal camera. A thermal camera targets wavelengths in the far-infrared region and is a camera that senses the heat emitted from an object and visualizes the temperature information.

[0069] Furthermore, comparing with the hardware shown in Figure 2, the navigation camera 150 corresponds to the first imaging device 310, the CPU 301 for image processing, the main memory 302, and the non-volatile memory 303. The payload camera 160 corresponds to the second imaging device 320, the CPU 301 for image processing, the main memory 302, and the non-volatile memory 303.

[0070] Next, the user interface in the client device will be described using Figures 5A and 5B. Figure 5A shows how a visible light image is displayed in the user interface of a client device. Figure 5B shows how a non-visible light image is displayed in the user interface of a client device.

[0071] As shown in Figure 5A, the display device 410 of the client device 200 has a display area 801 for displaying images captured by the navigation camera 150 and a display area 802 for displaying images captured by the payload camera 160. Here, display area 802 is displayed superimposed on the left shoulder of display area 801. In Figure 5A, display area 801 displays a visible light image (VISIBLE) captured by the first visible light imaging mechanism 311a of the navigation camera 150. Display area 802 displays a visible light image (VISIBLE) captured by the second visible light imaging mechanism 321a of the payload camera 160. In this example, display area 802 shows a subject 803 that is partially hidden by surrounding trees.

[0072] Furthermore, controllers 420 for operating the drone 10 and inputting information are located on either side of the display device 410.

[0073] In Figure 5A, the controller 420, display device 410, and client device 200 are shown as an integrated unit, but they may also be configured separately, and the functions of the operation input unit 205 may be realized by a touch panel. Also, although an example of superimposing display area 802 on display area 801 is shown here, the right half of the display device 410 may be divided into display area 801 and the left half into display area 802. In this way, the size and arrangement of each display area can be arbitrarily changed.

[0074] Furthermore, as shown in Figure 5B, the display area 801 displays a non-visible light image (VISIBLE) captured by the first visible light imaging mechanism 311a of the navigation camera 150, similar to Figure 5A. However, the display area 802 superimposed on the display area 801 displays a non-visible light image (THERMAL) captured by the second non-visible light imaging mechanism 321b of the payload camera 160. In this example, the display area 802 shows the image of a thermally detected subject 803.

[0075] Here, Figures 5A and 5B show an example in which a visible light image is displayed in the display area 801, but a non-visible light image captured by the first non-visible light imaging mechanism 311b may also be displayed. Furthermore, the number of images displayed is not limited, and any image from among those captured by the first visible light imaging mechanism 311a, the first non-visible light imaging mechanism 311b, the second visible light imaging mechanism 321a, and the second non-visible light imaging mechanism 321b may be displayed.

[0076] In the display area 802 of Figure 5A, a visible light image is displayed, allowing the user to identify the clothing and skin color of the subject 803. Furthermore, the person can be identified from the shape and size of their eyes and nose. However, in a visible light image, if obstructions such as grass or leaves are visible in front of the subject 803, it may become difficult for the user to see the subject 803.

[0077] In contrast, in the display area 802 of Figure 5B, a non-visible light image (thermal image) is displayed, so the difference between the subject 803, which is a heat source (person), and the background, which is not a heat source, becomes larger, making it easier for the user to find the subject 803. Also, even if obstructions such as grass or leaves are reflected in front of the subject 803, these objects do not register heat, so visibility is not significantly reduced. However, unlike visible light images, the colors of the image cannot be distinguished in a non-visible light image. Furthermore, it becomes difficult to identify people or distinguish facial features. Thus, visible light images and non-visible light images are each suited to different types of scenes where identification is easier.

[0078] In the example above, we described a case where the non-visible light image is an infrared image (thermal image). However, the non-visible light image is not limited to infrared images (thermal images); it may also be an ultraviolet image or other image, as long as it is outside the wavelength range of visible light.

[0079] Next, we will explain the process by which the drone automatically tracks a subject using Figures 6 to 8. Figure 6 illustrates how a drone automatically tracks a subject. Figure 7A shows an example of how images captured during automatic tracking are displayed (part 1). Figure 7B shows an example of how images captured during automatic tracking are displayed (part two).

[0080] Figure 8 is a flowchart showing the process by which a drone automatically tracks a subject.

[0081] First, let's explain the concept of automatic tracking for drones. Automatic tracking (auto-follow mode) for drones is a function that allows the drone to automatically follow a specific object or person (subject). By using the automatic tracking function of a drone, the user (photographer) does not need to adjust the camera's movement themselves, making it easier to capture more dynamic footage.

[0082] For automatic tracking, the imaging system 100 of the drone 10 analyzes the captured images and calculates the position of the subject 803 within the images. Based on this calculation, the control unit 101 of the drone 10 tracks the subject 803 by driving the imaging drive unit 140 and the movement drive unit 501 to a position where the subject 803 is within the field of view (within the image) being captured.

[0083] Next, we will use Figures 6, 7A, and 7B to illustrate the automatic tracking process of the drone with concrete images. Figure 6 shows the drone 10 automatically tracking the subject 803, and how the subject 803 moves from the position of subject 811 before the move to the position of subject 812 after the move. The drone 10 controls the imaging drive unit 140 to perform a pan-tilt operation to position the subject 803 at the center of the field of view, and continues to capture the subject with the payload camera 160. At this time, the subject 811 before the move is partially hidden by the trees in the background from the perspective of the drone 10, while the subject 812 after the move is in a position where it can be directly seen by the drone 10.

[0084] As shown in Figure 7A, the subject 811 before it moved, which was photographed while the drone 10 was automatically tracking it, is displayed on the display device 410 as a visible image where it is partially hidden by the trees in the background from the perspective of the drone 10. Similarly, the subject 812 after it moved, which was photographed while the drone 10 was automatically tracking it, is displayed on the display device 410 as a visible image where it is in a position that can be directly seen from the perspective of the drone 10, as shown in Figure 7B.

[0085] In this way, the automatic tracking of the drone 10 makes it possible to keep the subject 803 within the field of view of the payload camera 160.

[0086] As demonstrated above, by acquiring images suitable for image analysis for automatic tracking, it becomes possible to reduce the chances of losing sight of the subject.

[0087] Next, we will explain the process of automatically tracking a subject using a drone, using Figure 8. First, the second image processing unit 123 of the imaging system 100 performs image analysis based on the pixel data of the visible light and invisible light images acquired by the captured payload camera 160, and performs edge detection (S900). Edge detection is one of the image processing techniques used when a drone's camera grasps a subject, and is a method for identifying contours and boundaries within an image. By capturing changes between the subject and the background, or between different objects, it emphasizes visual features and supports the recognition and tracking of objects. Here, an edge is the contour or boundary of an object, and is perceived as a part where abrupt changes in brightness or color occur within the image. This edge detection allows the drone 10 to accurately detect the contour of a subject and track its movement when automatically tracking the subject. There are various known algorithms for edge detection, such as the Sobel filter, Canny edge detection, and Laplacian filter.

[0088] In this case, the image analyzed for edge detection can be either a visible light image or a non-visible light image. Furthermore, if the image displayed in the display area 802 of Figure 5A and the image analyzed for edge detection are the same, the load on the imaging system 100 can be reduced by stopping the imaging mechanism that captures images that are neither displayed nor analyzed.

[0089] Next, the imaging system 100 transmits the visible light image or invisible light image detected by edge detection in S900 to the client device 200 via the network 20, and the display device 410 of the client device 200 displays the edge detection results for the visible light image and invisible light image (S901).

[0090] Next, the user looks at the image displayed in the display area 802 and, if there is a subject 803 that they want to focus on, selects (clicks) that subject 803 to instruct the system to automatically track that subject 803 (S902).

[0091] Next, the imaging system 100 receives information about the subject specified in S902 and an automatic tracking instruction from the user via the network 20 (S903).

[0092] Next, the imaging system 100 calculates an image analysis (edge ​​detection) score (evaluation value) for the visible light image and the invisible light image of the selected subject 803 (S904). In step S903, the edge detection score is calculated, for example, by contrast. The greater the difference between the subject 803 and the background, the higher the edge detection score. In the case of a visible light image, the score is higher if there is a difference in brightness between the subject 803 and the background. Conversely, the score will be lower if there is no difference in brightness or color between the subject 803 and the background. Furthermore, the score will be lower if part of the subject 803 is hidden by an obstruction such as grass or leaves in front of the subject 803 (between the subject 803 and the payload camera 160), as shown in Figure 7A. In the case of an invisible light image, the score will be higher if there is a difference in temperature between the subject 803 and the background. Conversely, the score will be lower if there is no difference in temperature between the subject 803 and the background. However, unlike visible light images, even if there are obstructions such as grass or leaves in front of subject 803 (between subject 803 and payload camera 160), subject 803 is less likely to score lower than in visible light images if the obstruction has high transmission characteristics in the infrared wavelength range being captured. Details of the score calculation will be explained later.

[0093] Next, the imaging system 100 compares the edge detection score obtained from the visible light image calculated in step S904 with the edge detection score obtained from the invisible light image (S905). Based on the comparison, it determines which image has a higher edge detection score. Hereafter, the image with the higher score will be referred to as the "analysis image". One of the imaging mechanisms capturing the analysis image (the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b of the second imaging device 320 in Figure 2) is selected as the imaging mechanism to be used for automatic tracking control.

[0094] Next, the imaging system 100 of the drone 10 calculates the motion vector of the subject 803 based on the identified and analyzed image in step S905 (S906). The motion vector is a quantity that has the direction and magnitude of the drone 10's movement and is calculated from the image difference between frames. That is, by comparing the edge detection results of consecutive frames, the amount and direction of edge displacement are calculated as the motion vector.

[0095] Next, the drone 10 controls the imaging drive unit 140 or the movement drive unit 501 to perform automatic tracking (S907) based on the movement vector calculated in step S906 and the position of the subject 803 within the field of view. At this time, if the subject 803 is off-center from the field of view, the imaging drive unit 140 or the movement drive unit 501 is controlled in a direction that allows it to be photographed at the center of the field of view. However, it is desirable to predict the movement of the subject 803 based on the movement vector and correct the direction and amount of drive so that the subject 803 can be photographed at the center of the field of view in the next frame. Also, if the drone 10 is moving, corrections are made based on the direction and amount of movement of the drone 10.

[0096] Next, we will discuss the advantages of using contrast difference around edges as the edge detection score and how to calculate that score.

[0097] As described in this embodiment, using the contrast difference around an edge as the edge detection score is an effective means of directly evaluating the clarity and sharpness of the edge. By scoring the contrast difference, it is possible to quantitatively evaluate how prominent an edge is and how well it is distinguishable from other parts. In other words, the contrast difference can be used as an indicator that reflects the differences in brightness and color around the edge.

[0098] First, the imaging system 100 performs edge detection on both the visible light image and the invisible light image. The edge detection algorithm was explained in the section on S900.

[0099] Next, the brightness change (for visible light images) and temperature difference (for non-visible light images) around the detected edge are measured, and the contrast difference is calculated. This is done by determining the gradient magnitude for each pixel using the following equation (Equation 1).

number

[0100] Here is the G x , G y These represent changes in brightness and temperature in the horizontal and vertical directions, respectively.

[0101] Next, the contrast differences calculated for each edge are combined to calculate the overall score. The score can be obtained, for example, in the following way: (1) The score is the average or sum of the gradient strength around the edge. (2) Calculate the local contrast and take the difference between the maximum and minimum values ​​as the score.

[0102] Next, because the image characteristics (resolution and dynamic range) differ, the scores are normalized to make them comparable. Possible normalization methods include dividing each image's contrast score by the maximum contrast score or the overall average contrast score. That is, the normalized contrast score is obtained using either (Equation 2) or (Equation 3) below.

number

[0103] Next, the normalized contrast scores of the visible light image and the invisible light image (infrared image) are compared. Typically, different results are obtained depending on the scene and environmental conditions, but the image with the higher score is considered to have clearer edges and is therefore more suitable for automatic tracking.

[0104] Thus, the imaging system 100 of this embodiment calculates the contrast score for edge detection of visible light images and non-visible light images in real time, enabling the drone 10 to automatically select the image with the higher score.

[0105] Next, we will explain the relationship between the displayed image and the analyzed image.

[0106] Here, the analysis image is the image used for automatic tracking, as defined above, and the display image is the image displayed in the display area 802 of the client device 200.

[0107] The analyzed image and the displayed image do not necessarily have to match.

[0108] The following describes variations in how the client device 200 switches the displayed image as appropriate when the drone shown in Figure 8 performs the process of automatically tracking a subject. These methods may be used individually or in combination, as long as the operations are not contradictory. Furthermore, the display switching control may be performed by the imaging system 100 of the drone 10 or by the client device 200. (1) A method to prevent the display from switching even if the displayed image and the analyzed image are different.

[0109] For example, if the displayed image is a visible light image and the analyzed image becomes a non-visible light image, the displayed image will remain as a visible light image without changing. This reduces the possibility of the user missing a crucial moment due to image switching. This also applies when the displayed image is a non-visible light image and the analyzed image is a visible light image. (2) Method for switching the display when the displayed image and the analyzed image are different.

[0110] In this case, since the analysis image has a higher edge detection score, the image not displayed to the user is likely to be easier to see. Therefore, the displayed image is changed to match the analysis image. For example, if the displayed image is a visible light image and the analysis image is a non-visible light image, the displayed image is changed to a non-visible light image. This can also be applied if the displayed image is a non-visible light image and the analysis image is a visible light image. In this case, it means that the display is not switched if the displayed image and the analysis image are the same. Therefore, since images with low scores will not be used as either the displayed image or the analysis image, power consumption can be reduced by limiting some functions in the imaging process, such as lowering the frame rate for shooting or distribution or increasing the image compression ratio. Alternatively, functions such as shooting, distribution, and recording may be stopped. (3) How to switch when the score difference is large

[0111] If the score difference is large, the analysis image is likely to be easier to view. Also, if the difference is small, it may be better not to switch, as shown in (1). Therefore, the switch should only be performed when the score difference is large. For example, if the displayed image is a visible light image and the analysis image is a non-visible light image, and the score difference between them is greater than a predetermined threshold, the displayed image should be changed to the non-visible light image. This can also be applied when the displayed image is a non-visible light image and the analysis image is a visible light image. (4) A method for displaying a visible light image and a non-visible light image in the display area when automatic tracking is started.

[0112] In this method, both visible light and invisible light images are displayed in the display area when the drone 10 starts automatically tracking the subject. This allows the user to obtain a lot of information about the subject 803 during automatic tracking.

[0113] Next, we will describe the user interface for switching displays on the client device using Figures 9A and 9B. Figure 9A shows an example of displaying visible light and invisible light images in the display area of ​​a client device. Figure 9B shows an example of a user interface for selecting an image to display when switching between different images.

[0114] As shown in Figure 9A, when the drone 10 starts automatic tracking of a subject, the display area 802 is assumed to simultaneously display a visible light image and a non-visible light image, as shown in Figure 9A. The display area 802 may be changed by user operation, and the display area 802 may be divided into separate sections for each image.

[0115] Here, the operation input unit 205 of the client device 200 displays a selection dialog 810 in which the user selects whether to switch the displayed image, as shown in Figure 9A. By displaying the selection dialog 810 to the user and explicitly allowing them to make a selection in this way, unintended switching by the user can be prevented. Furthermore, when switching the displayed image in the method described in (3) above when the score difference is large (when it exceeds a certain threshold), the image can be switched immediately to an image that is easy for the user to see by forcibly switching without displaying the selection dialog 810.

[0116] Next, we will explain how to update the score used to evaluate images for automatic tracking.

[0117] The image evaluation scores used for automatic tracking, as described above, change with environmental conditions, so it may be desirable to update them.

[0118] The following describes variations for updating the score. These methods may be performed individually or in combination. (1) Method for recomparing scores after a certain period of time has elapsed

[0119] The imaging system 100 of the drone 10 acquires images, for example, every 10 seconds during automatic tracking, calculates scores for both visible light and invisible light images, and then re-compares them. (2) Method for recomparing scores when the amount of movement of the subject exceeds the threshold.

[0120] In this method, if the amount of movement of the subject exceeds a threshold, scores for both the visible light image and the invisible light image are calculated and re-compared. For example, the scores are re-compared if the rotation angle of the pan-tilt mechanism exceeds 5 degrees. Alternatively, distance information may be calculated from the distance acquisition unit 505 to calculate the movement distance of the subject 803. Alternatively, it may be calculated from the percentage of pixels that have moved within the field of view. (3) Method for recomparing when the surrounding environment in which the image is taken changes based on the analysis results of the image.

[0121] In this method, the imaging system 100 of the drone 10 analyzes a visible light image or a non-visible light image to calculate a histogram. A histogram is a data structure that associates the pixel values ​​of an image with their frequency of occurrence. When there is a change in the histogram, it is assumed that the surrounding environment has changed, and scores for the visible light image and the non-visible light image are calculated and re-compared. Alternatively, the histogram of the region in the direction of travel may be calculated from the movement vector calculated in step S906 of Figure 8. For example, in the example shown in Figure 6 where the drone is automatically tracking a subject, the location of subject 811 before movement is surrounded by trees, while the location of subject 812 after movement is not surrounded by trees, resulting in different histograms. Also, since the location of subject 811 before movement is surrounded by trees, the score of the non-visible light image is assumed to be higher. In this case, the location of subject 812 after movement is an open area without trees, so the score of the visible light image may be higher. Therefore, when subject 803 moves to the position of subject 812 after the initial movement, the scores will be recompared.

[0122] Alternatively, a histogram of the predicted position of subject 803, calculated from the movement vector, can be calculated and compared with the histogram of the current position of subject 803. This makes it possible to switch the analysis image before subject 803 moves. (4) Method for recomparing scores if the score decreases

[0123] If the score of the analyzed image is low, edges may not be detected properly, which can cause automatic tracking of the subject to malfunction. Therefore, when the score of the analyzed image is low, the imaging system 100 of the drone 10 can analyze the visible light image or the non-visible light image to calculate a score and re-compare it to select an appropriate image for analysis.

[0124] Next, we will describe the variations in the configuration of the imaging unit in the imaging system.

[0125] In this embodiment, an example was described in which each imaging unit has one imaging unit capable of capturing visible light images and one imaging unit capable of capturing invisible light images. Furthermore, it is necessary that at least a portion of the imaging areas of these imaging units overlap. For this reason, in this embodiment, an example was described in which the two imaging units of the payload camera 160 capture visible light images and invisible light images, respectively. However, it is not limited to this, and for example, the first visible light imaging unit 111 and the first invisible light imaging unit 112 of the navigation camera 150 may capture images for calculating the score for automatic tracking. Also, if the imaging angles of the navigation camera 150 and the payload camera 160 overlap, a combination of the first visible light imaging unit 111 and the second invisible light imaging unit 122, or a combination of the first invisible light imaging unit 112 and the second visible light imaging unit 121 may be used.

[0126] The imaging unit that performs image analysis and the imaging unit that performs automatic tracking may be different. For example, the image acquired by the first visible light imaging unit 111 or the first non-visible light imaging unit 112 of the navigation camera 150 may be used as the analysis image to drive the pan-tilt mechanism of the payload camera 160. Alternatively, instead of the pan-tilt mechanism of the payload camera 160, the drone 10 itself may be driven. In that case, the position acquisition unit 502, speed acquisition unit 503, attitude acquisition unit 504, distance acquisition unit 505 of the drone 10, and the drive mechanism of the movement drive unit 501 are driven based on the results of the image analysis.

[0127] Next, we will explain the variations in the modes of automatic tracking operation. In the above explanation, the tracking operation for the subject was shown as being performed by the imaging drive unit 140 and the movement drive unit 501. However, the imaging system 100 may also change the magnification by optical zoom or electronic zoom when capturing the subject. That is, if the subject 803 moves away from the center of the field of view, the magnification is changed to the wide-angle side to control the subject 803 so that it fits within the shooting field of view. Alternatively, the cropping range and position of the electronic zoom may be changed to match the position of the subject 803 within the field of view using digital processing.

[0128] Furthermore, although this embodiment describes a technique for centering automatic tracking within the field of view, it is not necessarily required to be centered. Alternatively, the technique could be to position the subject 803 within a specific range within the field of view.

[0129] Next, we will explain variations in the image analysis methods used to calculate the score. The above example describes calculating a score for selecting images for automatic tracking by calculating an edge detection score for the subject. However, the method is not limited to this; it may also be determined by feature point detection or the degree of agreement with training data. For example, attribute recognition may also be used. Attribute information such as body size and gender may be calculated from the feature points of the image, and a score may be calculated from that attribute information.

[0130] Additionally, person (face / body type) recognition, which determines whether an image represents a specific person based on its feature points, may be used to calculate the score.

[0131] Furthermore, the user may pre-train the system with target data to be tracked. Based on this training, a score is calculated from the degree of agreement between the training data and the acquired image. The system may also determine whether the subject is moving. For example, if there is a change in brightness, the system may determine that the subject is moving. In this case, if the drone 10 is moving, the position is corrected by the amount of pixels that have shifted due to the movement.

[0132] If the system is intended to track people, it will calculate a score to determine whether or not subject 803 is a person. This score may be, for example, a score (degree of agreement) calculated by comparing it with training data using machine learning.

[0133] Furthermore, different scoring methods may be used for visible light cameras and invisible light cameras. For example, the score for a visible light image may be calculated by detecting the edges (contrast) between the subject 803 and the background, while the score for an invisible light image may be calculated by the temperature difference between the subject 803 and the background.

[0134] Furthermore, while a person was used as an example of attribute recognition here, animals, cars (machines), etc., could also be used. Here, methods other than edge detection were shown, but this is not the only method, and it is possible to combine or modify them with other image analysis methods.

[0135] Furthermore, for image analysis requiring high resolution, such as in person recognition, visible light images, which allow for the acquisition of high-resolution images, are advantageous.

[0136] Next, we will explain the variations of subjects that are automatically tracked using Figures 10A and 10B. Figure 10A shows an example of a visible light image when the subject is an animal. Figure 10B shows an example of a visible light image when the subject is a vehicle.

[0137] In this embodiment, the subject 803 is given as an example of a person, but it is not limited to this, and may be other objects of a suitable size that the drone 10 can automatically track. For example, as shown in Figure 10A, the subject 821 may be an animal, and as shown in Figure 10B, the subject 822 may be a vehicle (a moving machine).

[0138] Next, we will explain the operation of the second visible light imaging mechanism and the second non-visible light imaging mechanism in relation to their respective imaging areas.

[0139] First, one can consider the case where the imaging areas of the second visible light imaging mechanism 321a and the second non-visible light imaging mechanism 321b are driven in conjunction.

[0140] In this case, the field of view and shooting range of the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b can be controlled in conjunction. When the pan-tilt mechanism of the imaging drive unit 140 is driven, the shooting range of both the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b is changed. Also, the field of view of the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b are the same, so when changing the zoom magnification, the zoom magnification of both the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b is changed.

[0141] Next, consider the case where the imaging areas of the second visible light imaging mechanism 321a and the second non-visible light imaging mechanism 321b are driven independently.

[0142] In this case, it is assumed that the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b each have a pan-tilt mechanism or zoom mechanism that can be driven independently. Therefore, if the second visible light imaging mechanism 321a and the second invisible light imaging mechanism 321b are imaging different shooting areas, it is desirable to change the shooting area of ​​either the second visible light imaging mechanism 321a or the second invisible light imaging mechanism 321b to make the shooting areas the same. However, the shooting areas do not necessarily have to be the same; at least some of the shooting areas can overlap, and the subject 803 can be photographed in that overlapping shooting area.

[0143] When the imaging areas are driven independently, the imaging unit not used for image analysis may perform complementary imaging by imaging a different imaging area. An example of complementary imaging is described below. Here, the image from the imaging unit not used for analysis will be referred to as the "complementary image." (1) Shooting techniques to prevent losing sight of the subject due to movement (Part 1)

[0144] To prevent losing sight of subject 803 due to its movement, the interpolation image is set to a wider angle than the analysis image. This allows for searching for subject 803 from a wider area image if it is lost. If the imaging system 100 loses sight of subject 803, it becomes possible to display the interpolation image or re-compare the scores according to the field of view of the interpolation image. (2) Shooting techniques to prevent losing sight of the subject due to movement (Part Two)

[0145] To prevent losing track of subject 803 as it moves, the target area is captured using a complementary image based on the movement vector analyzed by the analyzed image. (3) Methods for photographing different shooting areas

[0146] For example, if a navigation camera 150 is unavailable, a wide-angle shot of the front of the drone 10 may be taken to serve the same purpose as the navigation camera 150. Alternatively, the sides, directly below, and rear of the drone 10 may be photographed to acquire information about the surroundings.

[0147] Next, using Figure 11, we will describe an example of another user interface in which the user selects a subject to be automatically tracked on the client device. Figure 11 shows a user interface in which the user selects a subject to be automatically tracked on the client device.

[0148] This relates to the process of selecting a subject performed by the user in S902 of Figure 8. The user operates the controller 420 to select the subject 803 to be tracked. At this time, the selection is made by operating the touch panel or cursor 831. If edge detection is performed before selection, a user interface that shows the detection result, such as the detection frame 830, may be displayed, as shown in Figure 11. Alternatively, edge detection may be performed after selection, and a user interface that shows the detection result, such as the detection frame 830, may be displayed. Furthermore, there may be a setting to perform automatic tracking without user operation when there is a heat source or a moving object. There may also be a setting to perform automatic tracking without user operation based on the results of person recognition or attribute recognition.

[0149] Next, we will explain a concrete example of displaying the edge detection score for a subject using Figures 12A and 12B. Figure 12A shows an example of displaying the edge detection score for a subject in a visible light image. Figure 12B shows an example of displaying the edge detection score for a subject in a non-visible light image.

[0150] In this example, the visible light image shown in Figure 12A yields a low score of 40 points, resulting in a score of 840a. In contrast, the invisible light image shown in Figure 12B transmits light through the surrounding grass and leaves of subject 803, allowing for proper edge detection and a high score of 90 points, resulting in a score of 840b. Therefore, in the case of the visible light image in Figure 12A and the invisible light image in Figure 12B, the analyzed image is the invisible light image in Figure 12B. Note that the upper limit of the edge detection score can be arbitrarily set, but in the above example, the upper limit of the edge detection score was set to 100 points.

[0151] Next, we will explain the offset used for score comparison. In step S905 of Figure 8, the process of comparing the edge detection scores of the subject is described, but an offset may be applied at this time. For example, if you want to prioritize the visible light image, you can add an offset of +10 points (out of a maximum of 100 points) to the visible light image score. This allows you to prioritize the visible light image as the analysis image if the score difference is within 10 points. You may also apply conditions to the offset. For example, since there are many dark shooting areas at night, the drone 10 can acquire time information from the client device 200 and, during nighttime hours, prioritize the non-visible light image, adding an offset of +10 points to the non-visible light image score.

[0152] As described above in this embodiment, when performing automatic tracking of a subject with a drone, the edge detection score of images captured by a camera having a visible light imaging mechanism and an infrared imaging mechanism is calculated, and automatic tracking is performed based on the image with the higher score. This reduces the chance of losing sight of the target subject and ensures reliable acquisition of the subject.

[0153] (Composition 1) An imaging system for capturing images used for automatic tracking of a subject by a moving object, The moving body, A visible light imaging unit that captures a visible light image, The system includes a non-visible light imaging unit that captures a non-visible light image in which at least a portion of the imaging area overlaps with the visible light imaging unit, A score is calculated for the visible light image captured by the visible light image capture unit and the invisible light image capture unit for the same subject. An imaging system characterized by selecting either the visible light imaging unit or the non-visible light imaging unit based on the results of a comparison of calculated scores, and selecting it as the imaging unit to be used for controlling the automatic tracking of a subject by the moving object.

[0154] (Configuration 2) The imaging system according to configuration 1, characterized in that the score is calculated from one of the following: edge detection, feature point detection, or degree of agreement with training data for the visible light image and the non-visible light image.

[0155] (Composition 3) The imaging system according to configuration 2, characterized in that the edge detection score is calculated based on the contrast difference between the edge of the subject and the area surrounding the edge.

[0156] (Composition 4) In the imaging unit used for controlling the automatic tracking of a subject by the aforementioned moving object, An imaging system according to any one of configurations 1 to 3, characterized by calculating the direction of movement and position of the subject.

[0157] (Composition 5) In the imaging unit used for controlling the automatic tracking of a subject by the aforementioned moving object, The imaging system according to configuration 4, characterized in that the direction of movement of the subject is calculated as a movement vector.

[0158] (Composition 6) The system includes a visible light imaging unit and an imaging drive unit that drives the non-visible light imaging unit. By controlling the automatic tracking of the subject, An imaging system according to any one of configurations 1 to 5, characterized in that the imaging drive unit is driven to a position that brings the subject to be automatically tracked within the field of view.

[0159] (Composition 7) The imaging system according to configuration 6, characterized in that the imaging drive unit drives the visible light imaging unit and the non-visible light imaging unit by a pan-tilt mechanism or a zoom mechanism.

[0160] (Composition 8) By controlling the automatic tracking of the subject by the aforementioned moving body, An imaging system according to any one of configurations 1 to 7, characterized in that the cropping range or position is changed to a position that fits the subject within the field of view.

[0161] (Composition 9) The imaging system according to any one of configurations 1 to 8, characterized in that the score for evaluating the image for automatic tracking of the subject is updated based on any of the following: the elapsed time of automatic tracking, the movement of the subject being automatically tracked, changes in the surrounding environment during automatic tracking, or a decrease in the calculated score.

[0162] (Composition 10) It has a display unit that displays the image captured by the aforementioned moving body, An imaging system according to any one of configurations 1 to 9, characterized in that the image displayed on the display unit is switched based on the result of using either the visible light imaging unit or the non-visible light imaging unit for automatic tracking of the subject, or based on a calculated score.

[0163] (Composition 11) An imaging system according to any one of configurations 1 to 10, characterized in that complementary imaging is performed by an imaging unit, either the visible light imaging unit or the non-visible light imaging unit, which is not used for automatic tracking of the subject.

[0164] (Composition 12) An imaging system according to any one of configurations 1 to 7, characterized in that the function of either the visible light imaging unit or the non-visible light imaging unit that is not used for automatic tracking of the subject is restricted.

[0165] (Composition 13) A mobile device that captures images used for automatic tracking of a subject, A visible light imaging unit that captures a visible light image, A visible light imaging unit that captures a visible light image, The system includes a non-visible light imaging unit that captures a non-visible light image in which at least a portion of the imaging area overlaps with the visible light imaging unit, A score is calculated for the visible light image captured by the visible light image capture unit and the invisible light image capture unit for the same subject. A mobile body characterized by selecting either the visible light imaging unit or the non-visible light imaging unit based on the results of a comparison of calculated scores, and selecting it as the imaging unit to be used for controlling the automatic tracking.

[0166] (Composition 14) Furthermore, it has a drive unit for movement, By controlling the automatic tracking of the subject, The moving body according to configuration 13, characterized in that the moving drive unit for movement is driven to a position that brings the subject to be automatically tracked within the field of view.

[0167] (Method 1) An imaging method using an imaging system that captures images for automatic tracking of a subject by a moving object, The moving body, A visible light imaging unit that captures a visible light image, The system includes a non-visible light imaging unit that captures a non-visible light image in which at least a portion of the imaging area overlaps with the visible light imaging unit, The control unit of the imaging system calculates a score for the visible light image captured by the visible light image unit and the invisible light image captured by the invisible light imaging unit for the same subject. The control unit of the imaging system selects either the visible light imaging unit or the non-visible light imaging unit based on the results of comparing the calculated scores. An imaging method characterized in that the control unit of the imaging system is selected as an imaging unit to be used for controlling the automatic tracking of a subject by the moving object.

[0168] (Program 1) A program that causes a computer to execute each step of the imaging method described in Method 1.

[0169] (Storage medium 1) A computer-readable storage medium containing a program that causes a computer to execute each step of the imaging method described in Method 1. [Explanation of Symbols]

[0170] 1...Drone system, 10...Drone, 20...Network, 150...Navigation camera, 160...Payload camera 100...Imaging system, 101...Control unit, 102...Communication unit, 103...Storage unit, 110...First imaging unit, 111...First visible light imaging unit, 112...First non-visible light imaging unit, 113...First image processing unit, 120...Second imaging unit, 121...Second visible light imaging unit, 122...Second non-visible light imaging unit, 123...Second image processing unit, 140...Imaging drive unit, 151...Score calculation unit, 152...Movement vector calculation unit, 500...Navigation system, 501...Movement drive unit, 502...Position acquisition unit, 503...Speed ​​acquisition unit, 504...Attitude acquisition unit, 505...Distance acquisition unit, 200...Client device, 201...Control unit, 202...Communication unit, 203...Display unit, 204...Display control unit, 205...Operation input unit, 206...Storage unit, 207...External device I / F unit, 310...First imaging device, 311a...First visible light imaging mechanism, 311b...First non-visible light imaging mechanism, 312a...Visible light imaging optical system, 313a...Visible light image sensor, 314a...Visible light lens, 312b...Non-visible light imaging optical system, 313b...Non-visible light image sensor, 314b...Non-visible light lens, 320...Second imaging device, 321a...Second visible light imaging mechanism, 321b...Second non-visible light imaging mechanism, 322a...Visible light imaging optical system, 323a...Visible light image sensor, 324a...Visible light lens, 322b...Non-visible light imaging optical system, 323b...Non-visible light image sensor, 324b...Non-visible light lens, 301...CPU, 302...Main memory, 303...Non-volatile memory, Imaging drive mechanism 330..., 340...Communication I / F device, 330...Imaging drive mechanism, 331...Motor, 332...Gear, 333...Pan-tilt mechanism, 334...Motor driver, 335...Encoder 401...CPU, 402...Main memory, 403...Non-volatile memory, 404...Display I / F device, 405...External device I / F device, 406...Communication I / F device, 407...Input / output I / F device, 410...Display device, 420...Controller

Claims

1. An imaging system for capturing images used for automatic tracking of a subject by a moving object, The moving body, A visible light imaging unit that captures a visible light image, The system includes a non-visible light imaging unit that captures a non-visible light image in which at least a portion of the imaging area overlaps with the visible light imaging unit, A score is calculated for the visible light image captured by the visible light image capture unit and the invisible light image capture unit for the same subject. An imaging system characterized by selecting either the visible light imaging unit or the non-visible light imaging unit based on the results of a comparison of calculated scores, and selecting it as the imaging unit to be used for controlling the automatic tracking of a subject by the moving object.

2. The imaging system according to claim 1, characterized in that the score is calculated from one of the following: edge detection, feature point detection, or degree of agreement with training data for the visible light image and the non-visible light image.

3. The imaging system according to claim 2, characterized in that the edge detection score is calculated based on the contrast difference between the edge of the subject and the area surrounding the edge.

4. In the imaging unit used for controlling the automatic tracking of a subject by the aforementioned moving object, The imaging system according to claim 1, characterized in that it calculates the direction of movement and position of the subject.

5. In the imaging unit used for controlling the automatic tracking of a subject by the aforementioned moving object, The imaging system according to claim 4, characterized in that the direction of movement of the subject is calculated as a movement vector.

6. The system includes a visible light imaging unit and an imaging drive unit that drives the non-visible light imaging unit. By controlling the automatic tracking of the subject, The imaging system according to claim 1, characterized in that the imaging drive unit is driven to a position that brings the subject to be automatically tracked within the field of view.

7. The imaging system according to claim 6, characterized in that the imaging drive unit drives the visible light imaging unit and the non-visible light imaging unit by a pan-tilt mechanism or a zoom mechanism.

8. By controlling the automatic tracking of the subject by the aforementioned moving body, The imaging system according to claim 1, characterized in that the cropping range or position is changed to a position that fits the subject within the field of view.

9. The imaging system according to claim 1, characterized in that the score for evaluating the image for automatic tracking of the subject is updated based on any of the following: the elapsed time of automatic tracking, the movement of the subject being automatically tracked, changes in the surrounding environment during automatic tracking, or a decrease in the calculated score.

10. It has a display unit that displays the image captured by the aforementioned moving body, The imaging system according to claim 1, characterized in that the image displayed on the display unit is switched based on the result of using either the visible light imaging unit or the non-visible light imaging unit for automatic tracking of the subject, or based on a calculated score.

11. The imaging system according to claim 1, characterized in that complementary imaging is performed by either the visible light imaging unit or the non-visible light imaging unit that is not used for automatic tracking of the subject.

12. The imaging system according to claim 1, characterized in that the function of either the visible light imaging unit or the non-visible light imaging unit that is not used for automatic tracking of the subject is restricted.

13. A mobile device that captures images used for automatic tracking of a subject, A visible light imaging unit that captures a visible light image, The system includes a non-visible light imaging unit that captures a non-visible light image in which at least a portion of the imaging area overlaps with the visible light imaging unit, A score is calculated for the visible light image captured by the visible light image capture unit and the invisible light image capture unit for the same subject. A mobile body characterized by selecting either the visible light imaging unit or the non-visible light imaging unit based on the results of a comparison of calculated scores, and selecting it as the imaging unit to be used for controlling the automatic tracking.

14. Furthermore, it has a drive unit for movement, By controlling the automatic tracking of the subject, The mobile body according to claim 13, characterized in that the mobile drive unit for movement is driven to a position that brings the subject to be automatically tracked within the field of view.

15. An imaging method using an imaging system that captures images for automatic tracking of a subject by a moving object, The moving body, A visible light imaging unit that captures a visible light image, The system includes a non-visible light imaging unit that captures a non-visible light image in which at least a portion of the imaging area overlaps with the visible light imaging unit, The control unit of the imaging system calculates a score for the visible light image captured by the visible light image unit and the invisible light image captured by the invisible light imaging unit for the same subject. The control unit of the imaging system selects either the visible light imaging unit or the non-visible light imaging unit based on the results of comparing the calculated scores. An imaging method characterized in that the control unit of the imaging system is selected as an imaging unit to be used for controlling the automatic tracking of a subject by the moving object.

16. A program for causing a computer to perform each step of the imaging method described in claim 15.

17. A computer-readable storage medium storing a program for causing a computer to perform each step of the imaging method described in claim 15.

Images