Imaging control device, imaging control method, and program

Abstract

The present invention improves performance in a detection or identification process performed by a computer, and improves visibility when a user views an image, by continuously capturing an object within the angle of view of an imaging device and controlling a drive unit so as to reduce any change in the position of the object in the image. According to the present invention, an estimation unit estimates the direction of an object with respect to the installation position of an imaging device, a first control unit controls the imaging direction of the imaging device toward the estimated direction of the object, a detection unit detects the object in a captured image from the imaging device, and, when the object is detected, a second control unit controls the speed of a change in the imaging direction of the imaging device in accordance with a change in the position of the object in the captured image.

Description

Imaging control device, imaging control method, and program

[0001] The present disclosure relates to an imaging control device, an imaging control method, and a program, and particularly to an imaging control device, an imaging control method, and a program for controlling the imaging direction or the change speed of the imaging direction of an imaging device.

[0002] In recent years, there has been a concern about an increase in cases where drones, such as those referred to as drones or UAVs (Unmanned Aerial Vehicles), illegally enter a flight restricted area (for example, the sky above and the surrounding areas of airports and important facilities) without permission, and technologies for detecting the flight of drones within a monitoring area have been studied. Also, even when drones are used for material transportation, it is desirable to be able to detect flying objects around a logistics base for departure / arrival management and collision prevention of drones.

[0003] Patent Document 1 describes that a flight object within a monitoring area is detected and tracked by a monitoring radio wave detection device. Also, Patent Document 1 describes that based on the trajectory of a flight object detected by the monitoring radio wave detection device, the suspiciousness of the flight object is determined, and an imaging device provided with a drive unit capable of controlling the imaging direction takes images while variably changing the imaging direction so that the flight object with a high degree of suspiciousness is projected onto the center of the screen.

[0004] Japanese Unexamined Patent Application Publication No. 2017 - 167870

[0005] In order to detect or identify a flight object on an image using a computer with artificial intelligence or the like, or for a user to confirm a flight object on an image, it is conceivable to control the imaging direction of the imaging device based on the position information of the flight object acquired from the monitoring radio wave detection device so that the flight object is projected onto the center of the screen as in Patent Document 1. Also, when the imaging device has already captured a flight object within the angle of view, it is also possible to control the imaging direction of the imaging device based on the position of the flight object on the image.

[0006] However, the drive unit that controls the shooting direction of the camera may not accept the next control until the movement to the specified angle is complete, resulting in repeated acceleration and deceleration. In this case, with the control method described above, the position of the aircraft on the image will not be stable, and visibility will decrease. Furthermore, if the aircraft's movement speed is high, it will be difficult to keep it within the shooting field of view.

[0007] This disclosure has been made in view of the above-mentioned problems, and its purpose is to improve the performance of computer-based detection or identification processing and the visibility of the image when viewed by the user, by continuously capturing an object within the field of view of the imaging device and controlling the drive unit so that the change in the position of the object on the image is small.

[0008] A shooting control device according to one aspect of the present disclosure includes: estimation means for estimating the direction of an object with respect to the installation position of the shooting device; first control means for controlling the shooting direction of the shooting device toward the estimated direction of the object; detection means for detecting the object on the captured image of the shooting device; and second control means for controlling the rate of change of the shooting direction of the shooting device in accordance with the change in the position of the object in the captured image when the detection means continuously detects the object over multiple frames.

[0009] In a shooting control method according to one aspect of the present disclosure, a computer estimates the orientation of an object relative to the installation position of the shooting device, controls the shooting direction of the shooting device toward the estimated orientation of the object, detects the object on the image captured by the shooting device, and, if the object is detected, controls the rate of change of the shooting direction of the shooting device in accordance with the change in the position of the object in the captured image.

[0010] A program according to one aspect of this disclosure causes a computer to perform the following processes: estimating the direction of an object relative to the installation position of the imaging device; controlling the imaging direction of the imaging device toward the estimated direction of the object; detecting the object on the image captured by the imaging device; and, if the object is detected, controlling the rate of change of the imaging direction of the imaging device in accordance with the change in the position of the object in the captured image.

[0011] According to one aspect of this disclosure, even when the object is in motion, the object can be detected with high accuracy in each frame of the video image captured by the camera.

[0012] This figure schematically shows an example of an aircraft tracking system equipped with a photographic control device according to one embodiment. This figure shows an example of the content that the photographic control device according to one embodiment displays on the display device of the aircraft tracking system. This is a block diagram showing the configuration of the photographic control device according to one embodiment. This is a flowchart showing the operation of the photographic control device according to one embodiment. This is a block diagram showing the configuration of the photographic control device according to one embodiment. This figure shows an example of the hardware configuration of the photographic control device according to one embodiment.

[0013] [Embodiment 1] Figure 1 shows an example of an aircraft tracking system in Embodiment 1. The aircraft tracking system 1 includes a radio wave detection device 100, a photography device 200, a photography control device 10, and a display device 300.

[0014] The radio wave detection device 100 is a device that searches for flying objects within the monitoring area using radio waves and transmits the detection results to the image capture control device 10. In Figure 1, the fan-shaped figure simulates the range of the radio wave detection device 100's search for flying objects. Examples of flying objects include unmanned aircraft that can be flown by remote control or autopilot, such as unmanned aircraft, unmanned rotary-wing aircraft, and unmanned airships, as well as aircraft flown by people using specific aerial equipment such as balloons, hang gliders, and paragliders, and flying creatures such as birds.

[0015] Examples of radio wave detection devices 100 include radar and passive radar. A radar transmits radio waves and receives reflected waves that have been reflected by an aircraft. Using the time from the transmission of the radio waves to the reception of the reflected waves, and the direction in which the reflected waves were received, it is possible to calculate the presence or absence of an aircraft, as well as the distance to the aircraft and its direction.

[0016] A passive radar, also known as a radio wave detection sensor, can determine the location of an aircraft by detecting radio waves used for communication between aircraft, such as unmanned aerial vehicles (UAVs), and ground stations. There are several types of radars and passive radars.

[0017] When radar or passive radar is used as the radio wave detection device 100, the type of radar or passive radar is predetermined by the system designer or others, taking into consideration various factors such as the size of the monitoring area, the type of aircraft to be detected, the type of facility being monitored by the monitoring system to which the aircraft tracking system 1 is applied, and its surrounding environment.

[0018] In embodiments 1 and 2 described later, the types of radars and passive radars used as the radio wave detection device 100 are not limited, so their explanation will be omitted.

[0019] The imaging device 200 is a device that photographs an aircraft detected by the radio wave detection device 100, and includes a drive unit 201 for controlling the shooting direction. The drive unit 201 may be a pan / tilt head whose base angle can be variably controlled by specifying the angle in the pan / tilt direction or by specifying the angular velocity.

[0020] In Figure 1, the triangular shape simulates the shooting range of the imaging device 200. The imaging device 200 may be one unit or multiple units, as shown in Figure 1; the number is not limited. If multiple aircraft are detected simultaneously by the radio wave detection device 100, the target to be photographed by the imaging device 200 may be determined based on predetermined rules, or different aircraft may be photographed by multiple imaging devices 200. The lens that determines the field of view of the imaging device 200 is predetermined by the system designer or the like, depending on the size of the monitoring area, the distance to the detected target, the size of the detected target, etc.

[0021] Lenses can be classified into three types based on their angle of view: standard lenses, wide-angle lenses, and telephoto lenses. A standard lens has an angle of view of about 45 to 50 degrees, which is said to be close to the human field of vision. A wide-angle lens has a wider angle of view than a standard lens, for example, 60 degrees or more. A telephoto lens has a narrower angle of view than a standard lens, for example, 30 degrees or less.

[0022] Furthermore, the imaging device 200 may have a zoom function (digital zoom or optical zoom) for adjusting the field of view. The imaging device 200 continuously captures images within the shooting range and transmits the captured data to the imaging control device 10.

[0023] Figure 3 is a block diagram showing the configuration of the imaging control device 10 according to this embodiment 1. As shown in Figure 3, the imaging control device 10 includes an estimation unit 11, a first control unit 12, a detection unit 13, and a second control unit 14. Each function of the imaging control device 10 may be implemented as software, or as hardware using hardware resources such as a computer (Figure 6).

[0024] The estimation unit 11 performs signal processing on the radio wave detection signal output from the radio wave detection device 100 and estimates the direction of the aircraft relative to the installation position of the imaging device 200 at the time of detection (step S101 in Figure 4). The estimation unit 11 may further estimate the relative angular velocity of the aircraft with respect to the imaging device 200.

[0025] The output of the estimation unit 11 may be an estimated result of the direction of the aircraft at the time of detection, an estimated result of the direction of the aircraft at the present time based on the relative angular velocity of the aircraft, or a predicted result of the direction of the aircraft at a future time, taking into account the current angle and maximum angular velocity of the drive unit 201. The radio wave detection signal may include, for example, information on the aircraft's position (latitude, longitude, altitude), speed, and detection time.

[0026] The estimation unit 11 may estimate the direction of the aircraft relative to the installation position of the imaging device 200 at a time later than the current time, based on the estimation result of the direction of the aircraft relative to the installation position of the imaging device 200 at the current time and information on the time required to control the imaging direction of the imaging device 200.

[0027] The first control unit 12 controls the drive unit 201 at its maximum angular velocity so that the shooting direction of the shooting device 200 is directed towards the direction of the flying object input from the estimation unit 11 (step S102).

[0028] Furthermore, if the discrepancy between the latest direction of the aircraft estimated by the estimation unit 11 and the shooting direction being controlled by the first control unit 12 is too large (for example, if the discrepancy exceeds a certain angle in terms of the field of view of the shooting device 200), it is possible that the aircraft being photographed by the shooting device 200 has switched to a different aircraft (for example, a bird) than the aircraft being detected. Therefore, the first control unit 12 may adjust the shooting direction based on the estimation result of the estimation unit 11 so that the aircraft being detected is in the shooting direction of the shooting device 200.

[0029] The detection unit 13 starts processing when the angle control of the drive unit 201 is completed. The detection unit 13 detects the aircraft on the image captured by the imaging device 200 (step S103). The detection method is not limited here.

[0030] In one example, the detection unit 13 selects frames to be processed at predetermined intervals from a series of sequentially captured images (video) obtained by the imaging device 200. If the detection unit 13 detects an aircraft continuously across multiple frames (Yes in step S104), the second control unit 14 increases or decreases the angular velocity of the drive unit 201 in accordance with the change in the position of the aircraft in the images (step S105).

[0031] For example, the second control unit 14 controls the flying object on the image to remain at a fixed position within a predetermined area. The method for determining the degree of increase or decrease in the angular velocity of the drive unit 201 is not limited, but it may be determined based on a PID (Proportional Integral Differential) control law so that the error between the value obtained by converting the position change of the flying object into relative angular velocity with respect to the imaging device 200 approaches zero.

[0032] The flow then returns to step S103. Also, if the detection unit 13 fails to detect an aircraft (No in step S104), the processing of the detection unit 13 is stopped and the process returns to step S101.

[0033] The display device 300 is a device that provides information by displaying characters and images on a screen. Here, the display device 300 receives detection results from the radio wave detection device 100 and detection results from the detection unit 13 of the shooting control device 10, and provides information about the aircraft to, for example, the user of the aircraft tracking system (monitoring system). The display device 300 may be, for example, a monitor connected to the shooting control device 10, or a display device such as a portable terminal carried by the user.

[0034] (Example of a screen displayed on the display device 300) Figure 2 shows an example of a screen displayed on the display device 300. In the example shown in Figure 2, the upper left of the screen displays a detection result image showing the position of an aircraft detected by the radio wave detection device 100, and a mark indicating the aircraft being photographed by the camera device 200 among the aircraft on the detection result image. The shape of the mark is not limited to that shown in Figure 2; it may be a graphic, a symbol, or a mark resembling an aircraft, and can be set as appropriate by the system designer or the like.

[0035] Furthermore, if there are multiple imaging devices 200, different marks may be set according to the identification number of each imaging device 200. The image captured by the imaging device 200 is displayed on the right side of the screen. In Figure 2, a frame is displayed superimposed on the image area of ​​the aircraft; this indicates that it is an aircraft to be detected.

[0036] Furthermore, the means by which the display device 300 provides information is not limited to the above, as long as it can inform the user that the target aircraft has been detected.

[0037] (Effects of this embodiment) According to the configuration of Embodiment 1, the aircraft tracking system 1 moves the shooting direction of the shooting device 200 toward the aircraft detected by the radio wave detection device 100 at the maximum angular velocity using the first control unit 12 and the drive unit 201, while the second control unit 14 sequentially increases or decreases the angular velocity of the drive unit 201. As a result, the angular velocity changes more smoothly compared to when the angle is specified and controlled each time, making it possible to continuously and stably photograph the aircraft. This improves the detection or identification processing of aircraft by the computer and the visibility when the user checks the captured images.

[0038] [Embodiment 2] Embodiment 2 of the present disclosure will be described below.

[0039] In this second embodiment, components common to the first embodiment are denoted by the same reference numerals as in the first embodiment, and their descriptions are omitted.

[0040] (Configuration of the shooting control device 10) Figure 5 is a block diagram showing the configuration of the shooting control device 10 according to this second embodiment. As shown in Figure 5, the shooting control device 10 further includes a tracking unit 25 in addition to the configuration of the first embodiment.

[0041] The tracking unit 25 works in conjunction with the detection unit 13 to track the target aircraft using the captured images. Various tracking methods have been proposed for tracking subjects using captured images, and are not limited to those proposed here, but one example is a method that uses a tracking model generated by artificial intelligence (AI). This tracking model is generated, for example, by training with two sets of training data: a local image extracted only from the region of the aircraft being tracked, and a whole image including the background.

[0042] The input information to the tracking model consists of, for example, two images: an image captured by the imaging device 200 and a local image of the aircraft obtained as the output of the previous tracking model. The output information from the tracking model includes, for example, the position information of the aircraft being tracked in the input image and a local image extracted from the region of the aircraft.

[0043] In this case, for the time-series continuous captured images (video) obtained by the imaging device 200, frame images to be processed are selected at predetermined frame intervals. In one example, the tracking unit 25 executes a tracking process for tracking the target flying object as described above with respect to the selected frame images.

[0044] (Effects of This Embodiment) According to the configuration of Embodiment 2, since it includes the same configuration as Embodiment 1, the same effects as Embodiment 1 can be achieved. Further, in Embodiment 2, the imaging control device 10 further includes a tracking unit 25, and can also acquire the trajectory information of the flying object from the captured images of the imaging device 200. Thereby, the imaging control device 10 can improve the detection accuracy of the detected flying object and increase the information that can be provided to the user when the flying object is detected.

[0045] [Other Embodiments] The present disclosure is not limited to Embodiment 1 or Embodiment 2, and can take various embodiments. For example, the detection unit 13 may further identify the type of the detected flying object and provide the identification result to the user through the display device 300. In the display device 300, for example, a mark corresponding to the type of the flying object can be superimposed and displayed at the detection position of the flying object. Further, the detection unit 13 further has a function of determining whether the detected flying object is a suspicious flying object whose flight within the monitoring area is not permitted, and when it is determined that the flying object is a suspicious flying object, an alert may be issued to a predetermined notification destination.

[0046] Further, the initial detection method of the flying object in the area is not limited to the detection by the radio wave detection device 100, and detection may be performed using physical phenomena other than radio waves, such as sound waves and light waves. Also, in the above-described embodiments, the case where the detection target is a flying object has been described, but the detection target may be an object moving on the ground or at sea.

[0047] [Other Embodiments: Hardware Configuration] Each component of the imaging control device 10 described in the above embodiments represents a functional unit block. Some or all of these components are realized by, for example, an information processing device as shown in FIG. 6. FIG. 6 is a block diagram showing an example of the hardware configuration of the information processing device.

[0048] As shown in FIG. 6, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111.

[0049] The CPU 111 expands the program (code) in this embodiment stored in the storage device 113 into the main memory 112 and executes these in a predetermined order to perform various operations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Also, the program in this embodiment is provided in a state stored in a computer-readable recording medium 120. Note that the program in this embodiment may be distributed on the Internet connected via the communication interface 117.

[0050] In addition, specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and a mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0051] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0052] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0053] (Note) Some or all of the above embodiments may also be described as follows, but are not limited to the following.

[0054] (Note 1) A camera control device comprising: estimation means for estimating the direction of an object relative to the installation position of the camera device; first control means for controlling the shooting direction of the camera device toward the estimated direction of the object; detection means for detecting the object on the image captured by the camera device; and second control means for controlling the rate of change of the shooting direction of the camera device in accordance with the change in the position of the object in the image captured when the detection means detects the object.

[0055] (Note 2) The imaging control device according to Note 1, characterized in that the estimation means estimates the direction of the object relative to the installation position of the imaging device based on the detection result of the object in the monitoring area by the radio wave detection device.

[0056] (Note 3) The imaging control device according to Note 2, characterized in that the estimation means estimates the direction of the object relative to the installation position of the imaging device at the current time, using the detection time of the radio wave detection device, the position information of the detected object, the position information of the imaging device, and the current time.

[0057] (Note 4) The imaging control device according to Note 3, wherein the estimation means estimates the direction of the object relative to the installation position of the imaging device at a time earlier than the current time, based on the estimation result of the direction of the object relative to the installation position of the imaging device at the current time and information on the time required to control the imaging direction of the imaging device.

[0058] (Note 5) The photographic control device according to any one of Notes 1 to 4, characterized in that the first control means controls the photographic direction of the photographic device at the maximum angular velocity toward the direction of the object.

[0059] (Note 6) The photographic control device according to any one of Notes 1 to 5, characterized in that the first control means controls the shooting direction of the photographic device by operating a pan-tilt head on which the photographic device is mounted, which is variably controllable in the pan-tilt direction.

[0060] (Note 7) The first control means is characterized in that, when the direction of the object estimated by the estimation means and the shooting direction of the shooting device are inconsistent, the shooting device is corrected based on the estimation result of the estimation means, as described in any one of Notes 1 to 6.

[0061] (Note 8) The photographic control device according to any one of Notes 1 to 7, characterized in that the second control means increases or decreases the rate of change of the photographic direction of the photographic device based on the direction of movement of the object on the photographic image of the photographic device.

[0062] (Note 9) The photographic control device according to any one of Notes 1 to 8, characterized in that the second control means controls the photographic device so that the object on the photographed image remains at a fixed position within a predetermined area.

[0063] (Note 10) The photographic control device according to any one of Notes 1 to 9, further comprising tracking means for tracking the object on the image captured by the photographic device.

[0064] (Note 11) A shooting control method comprising: a computer estimating the direction of an object relative to the installation position of the shooting device; controlling the shooting direction of the shooting device toward the estimated direction of the object; detecting the object on the image captured by the shooting device; and, if the object is detected, controlling the rate of change of the shooting direction of the shooting device in accordance with the change in the position of the object in the captured image.

[0065] (Note 12) A program to cause a computer to perform the following: a process of estimating the direction of an object relative to the installation position of the imaging device; a process of controlling the imaging direction of the imaging device toward the estimated direction of the object; a process of detecting the object on the image captured by the imaging device; and, if the object is detected, a process of controlling the rate of change of the imaging direction of the imaging device in accordance with the change in the position of the object in the image captured.

[0066] Furthermore, some or all of the configurations described in Appendices 2 to 10, which are dependent on Appendice 1 (e.g., device) as described above, may also be dependent on Appendice 11 (e.g., method) and Appendice 12 (e.g., program) in the same dependent relationship as Appendices 2 to 10. Moreover, within the scope that does not deviate from each of the embodiments described above, some or all of the configurations described as appendices may also be dependent on various hardware, software, various recording means for recording software, or systems.

[0067] The present disclosure has been described above with reference to several embodiments. However, the present disclosure is not limited to the embodiments described above. Each embodiment can be combined with other embodiments as appropriate. Furthermore, various modifications to the configuration and details of the embodiments can be made as understandable to those skilled in the art within the scope of the present disclosure. This application claims priority based on Japanese Patent Application No. 2024-181954, filed on 17 October 2024, and incorporates all of its disclosures herein.

[0068] This disclosure can be used, for example, in an aircraft tracking system that uses a telephoto camera to detect and track unauthorized aircraft in no-fly zones, such as logistics drones.

[0069] 1. Aircraft tracking system 10. Image capture control device 11. Estimation unit 12. First control unit 13. Detection unit 14. Second control unit 25. Tracking unit 100. Radio wave detection device 200. Image capture device 201. Drive unit 300. Display device

Claims

1. A camera control device comprising: estimation means for estimating the direction of an object relative to the installation position of the camera device; first control means for controlling the shooting direction of the camera device toward the estimated direction of the object; detection means for detecting the object on the image captured by the camera device; and second control means for controlling the rate of change of the shooting direction of the camera device in accordance with the change in the position of the object in the image captured when the detection means detects the object.

2. The imaging control device according to claim 1, characterized in that the estimation means estimates the direction of the object relative to the installation position of the imaging device based on the detection result of the object in the monitoring area by the radio wave detection device.

3. The imaging control device according to claim 2, characterized in that the estimation means estimates the direction of the object with respect to the installation position of the imaging device at the current time, using the detection time of the radio wave detection device, the position information of the detected object, the position information of the imaging device, and the current time.

4. The imaging control device according to claim 3, wherein the estimation means estimates the direction of the object with respect to the installation position of the imaging device at a time earlier than the current time, based on the estimation result of the direction of the object with respect to the installation position of the imaging device at the current time and information on the time required to control the imaging direction of the imaging device.

5. The imaging control device according to any one of claims 1 to 4, characterized in that the first control means controls the imaging direction of the imaging device at the maximum angular velocity toward the direction of the object.

6. The photographic control device according to any one of claims 1 to 5, characterized in that the first control means controls the shooting direction of the photographic device by operating a pan-tilt head on which the photographic device is mounted, which is variably controllable in the pan-tilt direction.

7. The first control means is characterized in that, if the direction of the object estimated by the estimation means and the shooting direction of the shooting device are inconsistent, the first control means corrects the shooting direction of the shooting device based on the estimation result of the estimation means, as described in any one of claims 1 to 6.

8. The photographic control device according to any one of claims 1 to 7, characterized in that the second control means increases or decreases the rate of change of the photographic direction of the photographic device based on the direction of movement of the object on the photographic image of the photographic device.

9. The photographic control device according to any one of claims 1 to 8, characterized in that the second control means controls the photographic device so that the object on the photographed image remains at a fixed position within a predetermined area.

10. The photographic control device according to any one of claims 1 to 9, further comprising tracking means for tracking the object on the captured image of the photographic device.

11. A shooting control method comprising: a computer estimating the orientation of an object relative to the installation position of the shooting device; controlling the shooting direction of the shooting device toward the estimated orientation of the object; detecting the object on the image captured by the shooting device; and, if the object is detected, controlling the rate of change of the shooting direction of the shooting device in accordance with the change in the position of the object in the captured image.

12. A program to cause a computer to perform the following: a process of estimating the direction of an object relative to the installation position of the imaging device; a process of controlling the imaging direction of the imaging device toward the estimated direction of the object; a process of detecting the object on the image captured by the imaging device; and, if the object is detected, a process of controlling the rate of change of the imaging direction of the imaging device in accordance with the change in the position of the object in the captured image.

Images