Control device, control method, and control program

Abstract

Provided are a control device, a control method, and a control program capable of improving tracking performance with respect to a subject.　A camera (110), a radar (130), and a turning device (120) are included in an imaging system (100). The turning device (120) is controlled on the basis of the results of analyzing images captured by the camera (110), and the turning device turns the camera (110). The camera (110) includes a processor (113). The processor (113) sets a range of the analysis in the image captured by the camera (110) on the basis of information related to the distance of a specific object acquired from the radar (130), and information related to the size of the specific object (subject) and / or information related to the speed thereof.

Description

Control device, control method, and control program

[0001] The present invention relates to a control device, a control method, and a control program.

[0002] Patent Document 1 describes a tracking imaging device that includes an imaging means that captures an image in the direction of an imaging optical axis, a driving means that drives the imaging means to change the direction of the imaging optical axis in order to track a target object in the captured image, an area setting means that sets a detection processing area in a partial area of ​​the captured image, and an object position detection means that processes an image of the set detection processing area and detects position information of the target object in the detection processing area, and the area setting means sets the detection processing area for the next imaging based on the position information of the detected target object.

[0003] Patent document 2 describes an information processing device that detects a moving object using a sensor, identifies a first direction from the sensor toward the moving object, moves the optical axis direction of the camera along the first direction while the camera takes an image, and performs image analysis of the captured image to detect the moving object.

[0004] Patent Document 3 describes a control device that records the size of a target to be tracked that appears in an image captured by an imaging device, and when the state where the target to be tracked can no longer be detected from the captured image changes, calculates a predicted size of the target to be tracked in the captured image after zooming out at a predetermined magnification based on the recorded size, and if the predicted size is larger than the predetermined size, zooms out at the predetermined magnification and searches for the target to be tracked in the captured image after zooming out.

[0005] Patent Document 4 describes an image processing device that, when the movement of an object is detected from an input image, automatically controls the zoom magnification so that the object does not move out of the viewing angle.

[0006] Japanese Patent Application Publication No. 2010-136095 International Publication No. WO2019 / 093297 Japanese Patent Application Publication No. 2017-046321 Japanese Patent Application Publication No. 2017-228492

[0007] One embodiment of the technique of the present disclosure provides a control device, a control method, and a control program that can improve the tracking performance of a subject.

[0008] (1) A control device included in an imaging system comprising an imaging device, a rotation device that is controlled based on the results of analysis of an image captured by the imaging device and rotates the imaging device, and a distance measuring device, and the control device comprises a processor, wherein the processor sets the range of the analysis in the captured image based on information regarding the distance to a specific object obtained from the distance measuring device and at least one of information regarding the size or information regarding the speed of the specific object.

[0009] (2) The control device according to (1), wherein the specific object is a specific subject included in the captured image.

[0010] (3) The control device according to (1) or (2), wherein the processor performs the analysis.

[0011] (4) The control device according to any one of (1) to (3), wherein the processor controls the turning device based on a result of the analysis.

[0012] (5) The control device according to any one of (1) to (4), wherein the distance measuring device is a device that acquires information about a direction.

[0013] (6) The control device according to any one of (1) to (5), wherein the distance measuring device is a radio wave detector.

[0014] (7) A control device according to (5) or (6), wherein the processor controls the rotation device based on the detection result of the specific object by the analysis, and when the detection result satisfies a first condition, controls the rotation device based on information about the direction.

[0015] (8) The control device according to (7), wherein the first condition is a condition regarding whether or not the specific object is detected.

[0016] (9) The control device according to (8), wherein the first condition is a condition in which the specific object is not detected.

[0017] (10) A control device according to any one of (1) to (9), wherein the processor sets the range of the analysis based on a first range based on information in the captured image relating to the distance to the specific object and information relating to the size of the specific object, and a second range based on information in the captured image relating to the distance to the specific object and information relating to the speed of the specific object.

[0018] (11) The control device according to (10), wherein the processor sets a range narrower than the first range and wider than the second range as the range of the analysis.

[0019] (12) The control device according to (11), wherein the processor sets the first range as the range of the analysis when the first range is narrower than the second range.

[0020] (13) The control device according to any one of (1) to (12), wherein the focal length of the imaging device is variable, and the processor controls the focal length based on the range of the analysis.

[0021] (14) The control device according to (13), in which the processor controls the focal length so that an imaging range of the imaging device is wider than a range of the analysis.

[0022] (15) The control device according to any one of (1) to (14), wherein the processor causes a display device to display an image obtained by cropping the captured image based on a result of the analysis.

[0023] (16) The control device according to (15), wherein the processor cancels the cropping and causes the display device to display the captured image when the specific object is not detected by the analysis.

[0024] (17) The control device according to any one of (1) to (16), wherein the processor notifies a user when the specific object is not detected by the analysis.

[0025] (18) The control device according to any one of (1) to (17), wherein the processor selects the analysis mode from among a plurality of analysis modes based on information relating to the distance of the specific object.

[0026] (19) The control device according to any one of (1) to (18), wherein the processor sets a range of the analysis based on an environment in which the image is captured by the image capturing device.

[0027] (20) The control device according to any one of (1) to (19), wherein the processor updates at least one of the information about the size or the information about the speed of the specific object used to set the range of the analysis based on the information about the distance of the specific object and the result of the analysis.

[0028] (21) A control device according to any one of (1) to (20), wherein, when information regarding the distance to the specific object has not been obtained from the distance measuring device, the processor sets the range of the analysis based on information regarding the distance to the specific object based on the results of the analysis and at least one of information regarding the size or information regarding the speed of the specific object.

[0029] (22) A control device according to any one of (1) to (21), wherein the processor updates at least one of information relating to the size or information relating to the speed of the specific object used to set the range of the analysis based on information acquired from the distance measuring device.

[0030] (23) A control device according to any one of (1) to (22), wherein the processor acquires correspondence information between the type of a specific object and at least one of information relating to the size or information relating to the speed of the specific object, and acquires at least one of information relating to the size or information relating to the speed of the specific object to be used in setting the range of the analysis based on the type of the specific object and the correspondence information.

[0031] (24) The control device according to any one of (1) to (23), wherein the processor sets a position of the range of the analysis based on a result of the analysis.

[0032] (25) The control device according to any one of (1) to (22), wherein the processor sets the position of the range of the analysis based on information acquired from the distance measuring device.

[0033] (26) A control method for an imaging system including an imaging device, a rotation device that is controlled based on the results of analysis of an image captured by the imaging device and rotates the imaging device, and a distance measuring device, wherein a processor of a control device included in the imaging system sets the range of the analysis in the captured image based on information regarding the distance to a specific object obtained from the distance measuring device and at least one of information regarding the size or information regarding the speed of the specific object.

[0034] (27) A control program for an imaging system including an imaging device, a rotation device that rotates the imaging device under control based on the results of analysis of an image captured by the imaging device, and a distance measuring device, the control program causing a processor of a control device included in the imaging system to execute a process of setting the range of the analysis in the captured image based on information regarding the distance to a specific object obtained from the distance measuring device and at least one of information regarding the size or information regarding the speed of the specific object.

[0035] According to the present invention, it is possible to provide a control device, a control method, and a control program that are capable of improving the tracking performance of a subject.

[0036] 1 is a diagram illustrating an example of the configuration of an imaging system 100 of an embodiment. FIG. 1 is a block diagram illustrating an example of the configuration of an optical system and an electrical system of a camera 110. FIG. 2 is a block diagram illustrating an example of an imaging range 60 and an analysis range 61 of the camera 110. FIG. 3 is a flowchart illustrating a first control example of the swivel device 120 based on analysis of a captured image. FIG. 4 is a flowchart illustrating a second control example of the swivel device 120 based on analysis of a captured image. FIG. 5 is a flowchart illustrating a first setting example of the analysis range. FIG. 6 is a flowchart illustrating a second setting example of the analysis range. FIG. 7 is a flowchart illustrating a third setting example of the analysis range. FIG. 8 is a flowchart illustrating an example of control of the focal length of the camera 110 in accordance with a change in the analysis range. FIG. 9 is a diagram illustrating an example of cropping and displaying a detected subject. FIG. 10 is a diagram illustrating an example of canceling the cropping and display of a subject. FIG. 11 is a diagram illustrating an example of a display when the subject is lost. FIG. 12 is a flowchart illustrating a first modified example of the second control of the swivel device 120 based on analysis of a captured image. FIG. 13 is a flowchart illustrating a first modified example of the third setting of the analysis range. FIG. 14 is a flowchart illustrating a second modified example of the second control of the swivel device 120 based on analysis of a captured image. Fig. 1 is a flowchart showing a first shift example for shifting the position of a determined analysis range. Fig. 2 is a flowchart showing a second shift example for shifting the position of a determined analysis range. Fig. 3 is a diagram showing an example of shifting an analysis range 61 relative to an imaging range 60 of a camera 110. Fig. 4 is a diagram showing an example of how a control program for imaging control is installed into a computer 19 from a storage medium in which the control program is stored.

[0037] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0038] <Imaging System of Embodiment> Fig. 1 is a diagram showing an example of the configuration of an imaging system 100 of an embodiment. As shown in Fig. 1, the imaging system 100 includes a camera 110, which is an example of the "imaging device" of the present invention, a rotation device 120, and a radar 130, which is an example of the "distance measuring device" of the present invention. The camera 110 includes a lens 111, a sensor 112, and a processor 113. The processor 113 includes an image cropping / resizing unit 113a, an object detection unit 113b, a rotation control unit 113c, a distance acquisition unit 113d, and an image analysis range determination unit 113e.

[0039] The camera 110 is a camera that monitors a subject such as a drone, a car, or a person. The rotation device 120 is a rotation device that can rotate (pan and tilt) the camera 110. The camera 110 is attached to the rotation device 120. The monitoring direction of the camera 110 changes as the rotation device 120 rotates.

[0040] The radar 130 is an external device separate from the camera 110, and detects specific objects present in the vicinity. The specific objects are, for example, specific subjects such as drones, cars, and people that the camera 110 is attempting to monitor. The radar 130 is a radio detector that emits radio waves and detects the reflected waves, thereby measuring the direction and distance of the target. The radar 130 periodically acquires information regarding the distance and direction of the specific object. The radar 130 is installed, for example, near the camera 110.

[0041] The lens 111 of the camera 110 outputs light captured by the camera 110 when the camera 110 captures an image to the sensor 112. The sensor 112 converts the light into an electrical signal to generate image data, and outputs the generated image data to the video cropping / resizing unit 113a. The lens 111 also outputs "focal length information" in the imaging information of the camera 110 to the video analysis range determination unit 113e. The focal length information is information related to the focal length set when the camera 110 captures an image. The focal length of the camera 110 is variable.

[0042] The radar 130 transmits "distance information" relating to the distance of the detected specific object to the distance acquisition unit 113d of the camera 110. The radar 130 also transmits "direction information" relating to the direction of the detected specific object to the rotation control unit 113c of the camera 110. The radar 130 transmits the distance information and direction information to the camera 110 in real time using a command (e.g., serial / TCP (Transmission Control Protocol)).

[0043] The distance acquisition unit 113d outputs the distance information received from the radar 130 to the video analysis range determination unit 113e. The video analysis range determination unit 113e determines an analysis range for analyzing the video in the captured image captured by the camera 110 based on the "focal length information" and the "distance information." The analysis range is the range that is analyzed in order to detect a specific subject included in the captured image captured by the camera 110. The video analysis range determination unit 113e outputs the determined analysis range to the video cropping / resizing unit 113a.

[0044] Based on the determined analysis range, the video cropping / resizing unit 113a crops out the analysis range from the captured image based on the image data from the sensor 112. The video cropping / resizing unit 113a also resizes the image of the cropped analysis range, for example, to display it larger. The video cropping / resizing unit 113a outputs the resized image, that is, the analysis image, to the subject detection unit 113b.

[0045] The subject detection unit 113b performs a detection process using, for example, machine learning to detect a specific subject from the analysis image. The turning control unit 113c controls the turning of the turning device 120 based on the detection result by the subject detection unit 113b and directional information from the radar 130.

[0046] The camera (imaging device) 110 is an example of a "control device" in the present invention. Note that the "control device" in the present invention may be a device other than a camera, such as the swivel device 120, or a personal computer (PC) that is communicatively connected to the camera 110, the swivel device 120, and the radar 130.

[0047] Furthermore, at least one of the functions of the processor 113 may be implemented by a processor of a device external to the camera (imaging device) 110 (for example, a PC communicably connected to the camera 110). For example, the rotation control unit 113c may be implemented by a processor of the external device. Alternatively, the distance acquisition unit 113d and the image analysis range determination unit 113e may be implemented by a processor of the external device.

[0048] <Configuration of Optical System and Electrical System of Camera 110> FIG. 2 is a block diagram showing an example of the configuration of the optical system and electrical system of the camera 110. As shown in FIG. 2, the camera 110 includes an optical system 15 and an image sensor 25. The image sensor 25 is located downstream of the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis OA of the optical system 15 from the target subject side (object side) to the light receiving surface 25A side (image side) of the image sensor 25. The lens group 15B includes an anti-vibration lens 15B1, a focus lens (not shown), a zoom lens 15B2, and the like. The zoom lens 15B2 is supported by a lens actuator 21 (described later) so as to be movable along the optical axis OA. The anti-vibration lens 15B1 is supported by a lens actuator 17 (described later) so as to be movable in a direction perpendicular to the optical axis OA.

[0049] By increasing the focal length with the zoom lens 15B2, the camera 110 is set to the telephoto side, thereby narrowing the angle of view (narrowing the imaging range). By decreasing the focal length with the zoom lens 15B2, the camera 110 is set to the wide-angle side, thereby widening the angle of view (widening the imaging range).

[0050] The optical system 15 may include various lenses (not shown) in addition to the objective lens 15A and the lens group 15B. The optical system 15 may also include an aperture. The positions of the lenses, lens group, and aperture included in the optical system 15 are not limited, and the technology of the present disclosure is valid even if they are located in positions different from those shown in FIG. 2 .

[0051] The vibration-proof lens 15B1 is movable in a direction perpendicular to the optical axis OA, and the zoom lens 15B2 is movable along the optical axis OA.

[0052] The optical system 15 includes lens actuators 17 and 21. The lens actuator 17 applies a force to the vibration-proof lens 15B1 that fluctuates in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. The lens actuator 17 is controlled by an OIS (Optical Image Stabilizer) driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 fluctuates in a direction perpendicular to the optical axis OA.

[0053] The lens actuator 21 applies a force to the zoom lens 15B2 to move it along the optical axis OA of the optical system 15. The lens actuator 21 is controlled by the lens driver 28. When the lens actuator 21 is driven under the control of the lens driver 28, the position of the zoom lens 15B2 moves along the optical axis OA. When the position of the zoom lens 15B2 moves along the optical axis OA, the focal length of the camera 110 changes.

[0054] In addition, if the outline of the captured image is, for example, a rectangle with a short side in the pitch axis direction and a long side in the yaw axis direction, the angle of view in the pitch axis direction is narrower than the angle of view in the yaw axis direction and is narrower than the angle of view of the diagonal.

[0055] By the optical system 15 configured in this manner, light representing the imaging area is focused on the light receiving surface 25A of the imaging element 25, and the imaging area is imaged by the imaging element 25. The imaging element 25 is an example of the "sensor 112" shown in FIG. 1.

[0056] Incidentally, vibrations imparted to the camera 110 include, if outdoors, vibrations caused by passing cars, vibrations caused by wind, vibrations caused by road construction, etc., and if indoors, vibrations caused by the operation of an air conditioner, vibrations caused by people entering and exiting, etc. Therefore, the camera 110 shakes due to the vibrations imparted to the camera 110 (hereinafter also simply referred to as "vibrations").

[0057] In this embodiment, "shake" refers to a phenomenon in which the image of a target subject on the light-receiving surface 25A of the image sensor 25 of the camera 110 fluctuates due to a change in the positional relationship between the optical axis OA and the light-receiving surface 25A. In other words, "shake" can also be said to be a phenomenon in which the optical image formed on the light-receiving surface 25A fluctuates due to the tilt of the optical axis OA caused by vibrations applied to the camera 110. The fluctuation of the optical axis OA means, for example, that the optical axis OA is tilted with respect to a reference axis (for example, the optical axis OA before the shake occurs). Hereinafter, shake caused by vibrations will also be simply referred to as "shake."

[0058] Shake is included in the captured image as a noise component and affects the image quality of the captured image. Therefore, in order to remove the noise component included in the captured image due to shake, the camera 110 is equipped with a lens-side shake correction mechanism 29, an image sensor-side shake correction mechanism 45, and an electronic shake correction unit 33, which are used to correct shake.

[0059] The lens-side shake correction mechanism 29 and the image sensor-side shake correction mechanism 45 are mechanical shake correction mechanisms that apply power generated by a drive source such as a motor (for example, a voice coil motor) to a shake correction element (for example, the vibration-proof lens 15B1 and / or the image sensor 25) to move the shake correction element in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake.

[0060] Specifically, the lens-side image stabilization mechanism 29 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image stabilization lens 15B1, thereby moving the image stabilization lens 15B1 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The image sensor-side image stabilization mechanism 45 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image sensor 25, thereby moving the image sensor 25 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The electronic image stabilization unit 33 corrects shake by performing image processing on the captured image based on the amount of shake. In other words, the image stabilization unit (image stabilization component) performs shake correction mechanically or electronically using a hardware and / or software configuration. Here, mechanical shake correction refers to shake correction achieved by mechanically moving shake correction elements such as the anti-vibration lens 15B1 and / or the image sensor 25 using power generated by a driving source such as a motor (e.g., a voice coil motor), and electronic shake correction refers to shake correction achieved, for example, by image processing performed by a processor.

[0061] As an example, as shown in FIG. 2, the lens-side shake correction mechanism 29 includes an anti-vibration lens 15B1, a lens actuator 17, an OIS driver 23, and a position sensor 39.

[0062] Various well-known methods can be used to correct shake using the lens-side shake correction mechanism 29. In this embodiment, the method used is to correct shake by moving the vibration-proof lens 15B1 based on the amount of shake detected by a shake amount detection sensor 40 (described later). Specifically, shake correction is performed by moving the vibration-proof lens 15B1 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0063] A lens actuator 17 is attached to the vibration-proof lens 15B1. The lens actuator 17 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the vibration-proof lens 15B1 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the lens actuator 17 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be used instead of the voice coil motor.

[0064] The lens actuator 17 is controlled by an OIS driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 is mechanically changed within a two-dimensional plane perpendicular to the optical axis OA.

[0065] The position sensor 39 detects the current position of the vibration-proof lens 15B1 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 39. Here, the current position of the vibration-proof lens 15B1 refers to the current position within a two-dimensional plane of the vibration-proof lens. The vibration-proof lens two-dimensional plane refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 39, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photo sensor, or the like may be used instead of the Hall element.

[0066] The lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 along at least one of the pitch axis direction and the yaw axis direction within the range that is actually captured. In other words, the lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 within the two-dimensional vibration-proof lens plane by an amount corresponding to the amount of shake.

[0067] The image sensor side shake correction mechanism 45 includes the image sensor 25 , a BIS (Body Image Stabilizer) driver 22 , an image sensor actuator 27 , and a position sensor 47 .

[0068] As with the shake correction method used by the lens-side shake correction mechanism 29, various well-known methods can be used to correct shake using the image sensor-side shake correction mechanism 45. In this embodiment, the shake correction method used is to correct shake by moving the image sensor 25 based on the amount of shake detected by the shake amount detection sensor 40. Specifically, shake correction is performed by moving the image sensor 25 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0069] An imaging element actuator 27 is attached to the imaging element 25. The imaging element actuator 27 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the imaging element 25 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the imaging element actuator 27 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be applied instead of the voice coil motor.

[0070] The imaging element actuator 27 is controlled by the BIS driver 22. When the imaging element actuator 27 is driven under the control of the BIS driver 22, the position of the imaging element 25 is mechanically moved in a direction perpendicular to the optical axis OA.

[0071] The position sensor 47 detects the current position of the image sensor 25 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 47. Here, the current position of the image sensor 25 refers to the current position within a two-dimensional plane of the image sensor. The two-dimensional plane of the image sensor refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 47, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0072] The camera 110 includes a computer 19, a DSP (Digital Signal Processor) 31, an image memory 32, an electronic shake correction unit 33, a communication I / F 34, a shake amount detection sensor 40, and a UI (User Interface) device 43. The computer 19 includes a memory 35, a storage 36, and a CPU (Central Processing Unit) 37. The processor 113 in FIG. 1 is provided within the computer 19, and is, for example, the CPU 37. However, the processor 113 is not limited to the CPU 37, and may be another processor provided in the camera 110.

[0073] The image sensor 25, DSP 31, image memory 32, electronic shake correction unit 33, communication I / F 34, memory 35, storage 36, CPU 37, shake amount detection sensor 40, and UI device 43 are connected to a bus 38. The OIS driver 23 is also connected to the bus 38. Note that, for convenience of illustration, in the example shown in Fig. 2, one bus is shown as the bus 38, but multiple buses may be used. The bus 38 may be a serial bus or a parallel bus such as a data bus, an address bus, and a control bus.

[0074] The memory 35 temporarily stores various types of information and is used as a work memory. An example of the memory 35 is a random access memory (RAM), but other types of storage devices may be used. The storage 36 stores various programs for the camera 110. The CPU 37 reads the various programs from the storage 36 and executes them on the memory 35 to control the entire camera 110. Examples of the storage 36 include a flash memory, an SSD, an EEPROM, and an HDD. Alternatively, various types of non-volatile memory, such as a magnetoresistive memory or a ferroelectric memory, may be used instead of or in combination with the flash memory.

[0075] The imaging element 25 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging element 25 captures an image of a target subject at a predetermined frame rate under the direction of the CPU 37. The "predetermined frame rate" here refers to, for example, several tens to several hundreds of frames per second. The imaging element 25 itself may have a built-in control device (image sensor control device), in which case the image sensor control device performs detailed control of the imaging element 25 in accordance with imaging instructions output by the CPU 37. The imaging element 25 may also capture an image of a target subject at a predetermined frame rate under the direction of the DSP 31, in which case the image sensor control device performs detailed control of the imaging element 25 in accordance with imaging instructions output by the DSP 31. The DSP 31 is sometimes referred to as an ISP (Image Signal Processor).

[0076] The light receiving surface 25A of the image sensor 25 is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the image sensor 25, each photosensitive pixel is exposed to light, and photoelectric conversion is performed for each photosensitive pixel. The electric charge obtained by photoelectric conversion for each photosensitive pixel is an analog image signal representing the target subject. Here, a plurality of photoelectric conversion elements (e.g., photoelectric conversion elements with color filters) sensitive to visible light are used as the photosensitive pixels. In the image sensor 25, a plurality of photoelectric conversion elements are used, including a photoelectric conversion element sensitive to R (red) light (e.g., a photoelectric conversion element with an R filter corresponding to R), a photoelectric conversion element sensitive to G (green) light (e.g., a photoelectric conversion element with a G filter corresponding to G), and a photoelectric conversion element sensitive to B (blue) light (e.g., a photoelectric conversion element with a B filter corresponding to B). In the camera 110, these photosensitive pixels are used to capture images based on visible light (e.g., light on the short wavelength side of approximately 700 nanometers or less). However, this embodiment is not limited to this, and images may be captured based on infrared light (e.g., light on the long wavelength side of approximately 700 nanometers). In this case, a plurality of photoelectric conversion elements sensitive to infrared light may be used as the photosensitive pixels. In particular, for capturing images of short-wavelength infrared (SWIR), an InGaAs sensor and / or a Type-II Quantum Well (T2SL: Simulation of Type-II Quantum Well) sensor may be used, for example.

[0077] The image sensor 25 performs signal processing such as A / D (Analog / Digital) conversion on the analog image signal to generate a digital image, which is a digital image signal. The image sensor 25 is connected to the DSP 31 via a bus 38, and outputs the generated digital image to the DSP 31 via the bus 38 in frame units.

[0078] Note that, although a CMOS image sensor is described here as an example of the image sensor 25, the technology of the present disclosure is not limited thereto, and a CCD (Charge Coupled Device) image sensor may also be applied as the image sensor 25. In this case, the image sensor 25 is connected to the bus 38 via an AFE (Analog Front End) (not shown) that has a built-in CCD driver. The AFE generates a digital image by performing signal processing such as A / D conversion on the analog image signal obtained by the image sensor 25, and outputs the generated digital image to the DSP 31. The CCD image sensor is driven by a CCD driver built into the AFE. Of course, the CCD driver may also be provided separately.

[0079] The DSP 31 performs various types of digital signal processing on the digital image. The various types of digital signal processing refer to, for example, demosaic processing, noise removal processing, gradation correction processing, color correction processing, etc. The DSP 31 outputs the digital image after digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0080] The shake amount detection sensor 40 is a device that includes, for example, a gyro sensor, and detects the amount of shake of the camera 110. In other words, the shake amount detection sensor 40 detects the amount of shake in each of a pair of axial directions. The gyro sensor detects the amount of rotational shake around each of the pitch axis, yaw axis, and roll axis (axis parallel to the optical axis OA). The shake amount detection sensor 40 detects the amount of shake of the camera 110 by converting the amount of rotational shake around the pitch axis and the amount of rotational shake around the yaw axis detected by the gyro sensor into the amount of shake in a two-dimensional plane parallel to the pitch axis and yaw axis.

[0081] Here, a gyro sensor is given as an example of the shake amount detection sensor 40, but this is merely an example, and the shake amount detection sensor 40 may also be an acceleration sensor. The acceleration sensor detects the amount of shake in a two-dimensional plane parallel to the pitch axis and yaw axis. The shake amount detection sensor 40 outputs the detected amount of shake to the CPU 37.

[0082] Furthermore, although an example in which the amount of shake is detected by a physical sensor, namely, the shake amount detection sensor 40, is given here, the technology of the present disclosure is not limited to this. For example, a motion vector obtained by comparing successive captured images stored in the image memory 32 in time series may be used as the amount of shake. Furthermore, the amount of shake that is ultimately used may be derived based on the amount of shake detected by the physical sensor and the motion vector obtained by image processing.

[0083] The CPU 37 acquires the amount of shake detected by the shake amount detection sensor 40, and controls the lens-side shake compensation mechanism 29, the image sensor-side shake compensation mechanism 45, and the electronic shake compensation unit 33 based on the acquired amount of shake. The amount of shake detected by the shake amount detection sensor 40 is used for shake compensation by each of the lens-side shake compensation mechanism 29 and the electronic shake compensation unit 33.

[0084] The electronic shake correction unit 33 is a device including an ASIC (Application Specific Integrated Circuit). The electronic shake correction unit 33 corrects shake by performing image processing on the captured image in the image memory 32 based on the amount of shake detected by the shake amount detection sensor 40.

[0085] Note that, although a device including an ASIC is exemplified as the electronic shake correction unit 33 here, the technology of the present disclosure is not limited to this. For example, the electronic shake correction unit 33 may be a device including a field programmable gate array (FPGA) or a programmable logic device (PLD). Furthermore, for example, the electronic shake correction unit 33 may be a device including multiple ASICs, FPGAs, and PLDs. Furthermore, a computer including a CPU, storage, and memory may be employed as the electronic shake correction unit 33. There may be a single CPU or multiple CPUs. Furthermore, the electronic shake correction unit 33 may be realized by a combination of hardware and software configurations.

[0086] The communication I / F 34 is, for example, a network interface, and controls the transmission of various information between the camera 110 and the rotation device 120 via a network. This network is, for example, a wide area network (WAN) such as the Internet, or a local area network (LAN). The communication I / F 34 performs communication between the camera 110 and the rotation device 120.

[0087] Here, the rotation device 120 rotates in accordance with control instructions (for example, pan control values ​​and tilt control values) output from the camera 110 via the communication I / F 34, but may also rotate so as to track the subject to be tracked based on metadata output from the camera 110, in addition to or instead of the control instructions from the camera 110. This allows the rotation device 120 to obtain more accurate information about the subject, thereby improving the subject tracking performance.

[0088] For example, when the rotation device 120 outputs a command to acquire metadata to the camera 110 , the camera 110 outputs the metadata to the rotation device 120 via the communication I / F 34 .

[0089] The metadata includes, for example, information indicating the size and position of the subject. The metadata may also include information indicating the speed of the subject. This information can be used to control the rotation (pan / tilt) by the rotation device 120.

[0090] The metadata may also include information indicating the type of subject, which can be used by the swivel device 120 to distinguish between subjects (e.g., a human and a drone).

[0091] The metadata may also include information indicating the subject's unique ID, which can be used by the rotation device 120 to identify individual subjects when there are subjects of the same type.

[0092] The metadata may also include a timestamp, which is the time corresponding to the frame in which the subject was detected. The timestamp can be used by the rotation device 120 to calculate the subject's speed, even if the time it takes for the camera 110 to analyze the image is not constant.

[0093] The metadata may also include a score. The score is information indicating the reliability of the subject detection result in the camera 110, and can be used to improve the robustness of the turning device 120 in tracking the turning subject.

[0094] The timing for outputting metadata from the camera 110 to the rotation device 120 is, for example, the timing when the state of the subject (position, size, speed, lost state) changes. Note that a dead zone may be provided for determining changes in the state of the subject (position, size, speed, lost state).

[0095] Furthermore, the metadata may be embedded in the video stream output from the camera 110 and output from the camera 110 to the rotation device 120. This associates the metadata with the timestamp of the video stream, so that the rotation device 120 can calculate the speed of the subject even if the timestamp is not included in the metadata when the time required for image analysis in the camera 110 is not constant.

[0096] The UI device 43 includes a reception device 43A and a display 43B. The reception device 43A is, for example, a hard key or a touch panel, and receives various instructions from the user. The CPU 37 acquires the various instructions received by the reception device 43A and operates in accordance with the acquired instructions.

[0097] The display 43B displays various information under the control of the CPU 37. Examples of the various information displayed on the display 43B include the contents of various instructions accepted by the accepting device 43A, captured images, and the like.

[0098] <Image capture range and analysis range of camera 110> Fig. 3 is a block diagram showing an example of an image capture range 60 and an analysis range 61 of the camera 110. As shown in Fig. 3, in the image capture range 60 captured by the camera 110, a range of a predetermined size for analyzing a moving object (e.g., drone 50) that is the tracking target is set as an analysis range 61.

[0099] The imaging range 60 is a range corresponding to the entire area of ​​the image generated by the imaging element 25 in Fig. 2 (sensor 112 in Fig. 1). The analysis range 61 is a range set by the processor 113 based on, for example, the distance to the drone 50, the size of the drone 50, the movement speed of the drone 50, etc. In this example, the analysis range 61 is set approximately in the center of the imaging range 60.

[0100] <First Control Example of Swivel Device 120 Based on Analysis of Captured Images> FIG. 4 is a flowchart showing a first control example of the swing device 120 based on analysis of captured images.

[0101] The processor 113 of the camera 110 acquires the captured image of the camera 110 generated by the sensor 112 (step S11).

[0102] Next, the processor 113 extracts an analysis range for detecting a specific subject (e.g., the drone 50) from the captured image acquired in step S11 (step S12). The analysis range is a range set in a process for determining the analysis range (e.g., the process in FIG. 6 described later).

[0103] Next, the processor 113 analyzes the image of the analysis range extracted in step S12 and performs subject detection processing, for example, using machine learning, to detect the drone 50 (step S13).

[0104] Next, the processor 113 controls the rotation of the rotation device 120 so that the position of the drone 50 detected within the analysis range is located at the center of the analysis range (step S14). In this example, the rotation device 120 is rotated manually or by other control so that the drone 50 to be tracked is located approximately in the center of the analysis range. The processor 113 repeatedly executes this process for each frame of the captured image, for example.

[0105] <Second Example of Control of Swivel Device 120 Based on Analysis of Captured Images> FIG. 5 is a flowchart showing a second example of control of the swing device 120 based on analysis of captured images.

[0106] First, the processor 113 of the camera 110 determines whether or not the tracking target subject was detected from the image captured by the camera 110 in the previous subject detection process (step S10). The condition of whether or not the subject was detected is an example of the "first condition" of the present invention.

[0107] If the target is detected in step S10 (step S10: Yes), the processor 113 proceeds to step S11 and executes the processes from step S11 to step S14. Note that the processes from step S11 to step S14 are the same as the processes from step S11 to step S14 described in FIG. 4, and therefore description thereof will be omitted.

[0108] On the other hand, if the target has not been detected in step S10 (step S10: No), the processor 113 acquires, from the radar 130, directional information relating to the specific object detected by the radar 130 (step S15). If the target has not been detected, the condition (1) is an example of the "first condition" of the present invention.

[0109] Next, the processor 113 proceeds to step S14, where it controls the turning device 120 based on the direction information relating to the specific object acquired from the radar 130. The processor 113 repeatedly executes this process for each frame of the captured image, for example.

[0110] <First Example of Setting the Analysis Range> FIG. 6 is a flowchart showing a first example of setting the analysis range.

[0111] The processor 113 of the camera 110 acquires distance information from the radar 130 regarding the distance of a specific object detected by the radar 130 (step S21).

[0112] Next, the processor 113 determines the analysis range based on the "distance information" of the specific object acquired in step S21 and the "size information" regarding the size of the specific subject (e.g., the drone 50) (step S22). The size of the subject may be a size that is set in advance depending on the type of subject, or may be a size that the user assumes and sets for each subject. However, the size does not need to be an exact value, and may be an approximate value that is typically assumed.

[0113] For example, in the imaging range 60, analysis range 61, and subject (drone 50) shown in Figure 3, if the distance from the camera 110 to the drone 50 is ObjectD (m), the size of the drone 50 is ObjectW2 (m), the limit performance of the image analysis of the camera 110 (the ratio of the size of the drone 50 that can be detected in relation to the analysis range 61) is DetectRate (%), the focal length of the camera 110 is ShootingD (mm), the size of the sensor 112 is SensorW3 (mm), and the number of horizontal pixels in the imaging range 60 is ImageW0, the number of pixels DetectW in the horizontal direction in the analysis range 61 can be calculated using the following formula.

[0114] DetectW=ImageW0 / (ObjectD / ShootingD×SensorW3)×(ObjectW2 / (DetectRate / 100))

[0115] (ObjectD / ShootingD×SensorW3) corresponds to the range (m) of the entire image at the subject position. (ObjectW2 / (DetectRate / 100)) corresponds to the analysis range (m) at the subject position. That is, the processor 113 determines the number of horizontal pixels DetectW in the analysis range 61 based on the distance to the drone 50, the size of the drone 50, and the performance limit of the camera 110, i.e., how small a drone 50 can be detected by video analysis. The performance limit of the camera 110 is the limit (upper limit) at which the drone 50 cannot be detected if the analysis range 61 becomes wider than this. For example, the performance limit is when the ratio of drones 50 to the analysis range 61 is approximately 5% to 10%.

[0116] Next, the processor 113 sets the analysis range determined in step S22, for example, in the memory 35 of the computer 19 (step S23). The processor 113 repeatedly executes this process at a cycle longer than the frame cycle of the captured image, for example. However, it may also be repeated at the frame cycle.

[0117] <Second Example of Setting the Analysis Range> FIG. 7 is a flowchart showing a second example of setting the analysis range.

[0118] The processor 113 of the camera 110 acquires distance information from the radar 130 regarding the distance of a specific object detected by the radar 130 (step S31).

[0119] Next, the processor 113 determines the analysis range based on the "distance information" of the specific object acquired in step S31 and the "speed information" regarding the speed at which the specific subject (e.g., the drone 50) is moving (step S32). The speed at which the subject is moving may be a speed that is set in advance depending on the type of subject, or may be a speed that the user sets based on the assumption for each subject. For example, the speed may be the maximum anticipated speed of the subject.

[0120] For example, in the imaging range 60, analysis range 61, and subject (drone 50) shown in Figure 3, if the distance from the camera 110 to the drone 50 is ObjectD (m), the speed of the drone 50 is ObjectSpeed ​​(m / s), the frame rate of the video analysis of the camera 110 is FPS (fps), the focal length of the camera 110 is ShootingD (mm), the size of the sensor 112 is SensorW3 (mm), and the number of horizontal pixels in the imaging range 60 is ImageW0, the number of horizontal pixels DetectW in the analysis range 61 can be calculated using the following formula.

[0121] DetectW=ImageW0 / (ObjectD / ShootingD×SensorW3)×(ObjectSpeed / FPS×2)

[0122] (ObjectSpeed / FPS×2) refers to the analysis range (m) at the subject position. That is, the processor 113 determines the number of horizontal pixels DetectW in the analysis range 61 from the distance to the drone 50, the speed of the drone 50, and the frame rate of the video analysis (the reciprocal of the processing time). If the drone 50 moves during one frame and falls outside the analysis range 61, it will no longer be possible to continue tracking the drone 50. Therefore, the analysis range 61 is determined by setting the distance from the center of the video in which the drone 50 moves during one frame to the edge of the angle of view as the limit range. The limit range refers to the limit (lower limit) beyond which it will no longer be possible to continue tracking the drone 50 if the analysis range 61 becomes narrower.

[0123] Next, the processor 113 sets the analysis range determined in step S32 in, for example, the memory 35 of the computer 19 (step S33). The processor 113 repeatedly executes this process at a cycle longer than, for example, the frame cycle of the captured image. However, it may also be repeated at the frame cycle.

[0124] As described above, the camera 110 (control device) of the embodiment sets the analysis range 61 in the imaging range 60 of the camera 110 based on information regarding the distance of a specific object detected by the radar 130 and at least one of information regarding the size or information regarding the speed of the moving body (drone 50) to be tracked.

[0125] With this configuration, it is possible to set the analysis range 61 to a size suitable for analyzing the video depending on the detection state of the subject to be tracked. Therefore, even if the subject is small and moves quickly, such as a drone 50, it is possible to accurately detect the subject, thereby improving the tracking performance for the subject.

[0126] Furthermore, the camera 110 can control the rotation of the rotation device 120 based on the result of the detection process of the subject in the set analysis range 61. This makes it possible to appropriately adjust the rotation direction of the rotation device 120, and further improve the tracking performance of the subject.

[0127] <Third Example of Setting the Analysis Range> FIG. 8 is a flowchart showing a third example of setting the analysis range.

[0128] The processor 113 of the camera 110 acquires distance information from the radar 130 regarding the distance of a specific object detected by the radar 130 (step S41).

[0129] Next, the processor 113 determines a first range as the first candidate analysis range based on the "distance information" of the specific object acquired in step S41 and the "size information" regarding the size of the specific subject (e.g., the drone 50) (step S42). The size of the subject is as described in the first setting example.

[0130] Next, the processor 113 determines a second range as a second candidate analysis range based on the "distance information" of the specific object acquired in step S41 and the "speed information" related to the moving speed of the specific subject (step S43). The moving speed of the subject is as described in the second setting example.

[0131] Next, the processor 113 compares the first range determined in step S42 with the second range determined in step S43, and determines whether the first range is wider than the second range (step S44).

[0132] In step S44, if the first range is wider than the second range (step S44: Yes), the processor 113 determines an analysis range for detecting a specific subject that is narrower than the first range and wider than the second range, for example, a range intermediate in size between the first and second ranges (step S45). However, the range determined as the analysis range is not limited to an intermediate range, and may be any range between the second range and the first range.

[0133] Next, the processor 113 sets the determined analysis range in, for example, the memory 35 of the computer 19 (step S47).

[0134] On the other hand, if the first range is not wider than the second range in step S44 (step S44: No), the processor 113 determines the first range as the analysis range (step S46). The processor 113 proceeds to step S47 and sets the determined analysis range. In other words, if the first range is not wider than the second range, the processor 113 determines the analysis range by prioritizing the restriction due to the size of the specific subject (first range).

[0135] In this way, by setting the analysis range by comparing and considering the first range, which is determined based on the size information of the subject and the limits of the detection performance of the camera 110, and the second range, which is determined based on the speed information of the subject and the processing time for the video analysis of the camera 110, it is possible to set an analysis range that is suitable for the performance of the camera 110.

[0136] <Control of Focal Length Based on Analysis Range> FIG. 9 is a flowchart showing an example of control of the focal length of the camera 110 in response to a change in the analysis range.

[0137] The processor 113 of the camera 110 determines whether the set analysis range has changed (step S51). For example, the processor 113 determines whether the analysis range has changed by comparing it with the range set in the previous analysis range setting process. If the analysis range has not changed (step S51: No), the processor 113 repeats the process of step S51.

[0138] If the analysis range has changed in step S51 (step S54: Yes), the processor 113 sets the focal length of the camera 110 based on the changed analysis range (step S52). The processor 113 sets the focal length so that the imaging range 60 of the camera 110 (see FIG. 3) is wider (for example, 1.5 times wider) than the changed analysis range 61.

[0139] Next, the processor 113 instructs the lens driver 28 (see FIG. 2) to change the focal length to the focal length set in step S52 (step S53).

[0140] Next, the processor 113 determines whether or not the control of the focal length has been completed (step S54). If the control of the focal length has not been completed (step S54: No), the processor 113 repeats the process of step S54. If the control of the focal length has been completed (step S54: Yes), the processor 113 ends this process.

[0141] In this way, by changing the focal length of the camera 110 in response to changes in the analysis range, a more suitable analysis range can be set. This process is repeatedly executed in parallel with the process of setting the analysis range. However, since this process takes time to control the focal length, it is repeatedly executed for multiple frames.

[0142] <Display of Specific Subject> Figure 10 is a diagram showing an example of cropping and displaying a detected subject. When the processor 113 of the camera 110 detects a specific subject (e.g., drone 50) by analyzing the captured image, the processor 113 may crop the image of the detected drone 50 and resize the image so that it is displayed larger on the display screen 71 of the display device. The display device that displays the image may be a display device provided in the camera 110 or an external display device. This makes it possible to make the detected specific subject even easier to see.

[0143] 11 is a diagram showing an example of canceling the cropped display of a subject. When the drone 50 detected through analysis of the captured image becomes undetected, the processor 113 may cancel the cropped display of the image of the drone 50 that was previously displayed and display it on the display screen 71 of the display device. For example, when the drone 50 moves out of the analysis range and becomes unable to be detected, the processor 113 cancels the cropped display of the drone 50. This widens the angle of view, making it easier to detect the lost drone 50.

[0144] <Notification When a Subject is Lost> FIG. 12 is a diagram illustrating an example of a display when a subject is lost. If the processor 113 is unable to detect a specific subject (e.g., the drone 50) in the analysis of the analysis range, it notifies the user of this. The notification may involve, for example, superimposing information indicating that the drone 50 has been lost on an image displayed on the display device. For example, as shown in FIG. 12 , if the detected drone 50 moves out of the analysis range and is lost along the way, the processor 113 notifies the user by displaying a message 72 such as "The tracking target has been lost" on the display screen 71 of the display device. The processor 113 may also notify the user by, for example, covering the image displaying the message 72 with a red frame 73. Furthermore, the processor 113 may cancel the cropping of the drone 50, as in the case described with reference to FIG. 11 .

[0145] <Modification (Part 1) of Control of Swivel Device 120 Based on Analysis of Captured Images> FIG. 13 is a flowchart showing a modification (Part 1) of the second control (FIG. 5) of the swing device 120 based on analysis of captured images.

[0146] As shown in FIG. 13, the processes from step S10 to step S12 are the same as the processes from step S10 to step S12 in the second control example described with reference to FIG.

[0147] Next, the processor 113 selects an object detection model to be used to detect the object (drone 50) based on the distance between the object in the analysis range and the camera 110 (step S13A). The distance between the object and the camera 110 is, for example, distance information acquired from the radar 130. The object detection model refers to multiple analysis modes prepared in advance for detecting the object, and multiple modes with different analysis processing loads and analysis accuracy are prepared. The object detection model is, for example, a machine learning model that detects the object using machine learning. For example, the longer the distance between the object and the camera 110, the higher the processing load and the higher the accuracy of the machine learning model selected. This makes it possible to maintain high object detection performance even when the distance to the object is far.

[0148] The processes from step S13B to step S15 in FIG. 13 are the same as the processes from step S13 to step S15 in the second control example described with reference to FIG. 5, and therefore will not be described again.

[0149] <Modification (1) of Setting the Analysis Range> FIG. 14 is a flowchart showing a modification (1) of the third setting of the analysis range (FIG. 8).

[0150] As shown in FIG. 14, the process in step S41 is the same as the process in step S41 in the third setting example described with reference to FIG.

[0151] Next, the processor 113 determines a first range as a first candidate analysis range based on the "distance information" of the specific object acquired in step S41, "size information" about the size of the specific subject (e.g., the drone 50), and "brightness information" about the imaging environment of the camera 110 (step S42A). For example, the darker the subject being imaged, the narrower the first range set as the analysis range is determined to be. The brightness information is acquired, for example, by brightness measured using the automatic exposure function of the camera 110. This makes it possible to improve the analysis accuracy even in dark environments, such as on bad weather days or at night.

[0152] The processes from step S43 to step S47 in FIG. 14 are the same as the processes from step S43 to step S47 in the third setting example described with reference to FIG. 8, and therefore will not be described again.

[0153] In this example, the first range is determined in step S42A including the "brightness information," but this is not limiting. For example, the first range may be determined in step S42A without including the "brightness information," and after the analysis range is determined in step S45 or step S46, the analysis range may be corrected according to the "brightness information."

[0154] <Modification (Part 2) of Control of Swivel Device 120 Based on Analysis of Captured Images> FIG. 15 is a flowchart showing a modification (Part 2) of the second control (FIG. 5) of the swing device 120 based on analysis of captured images.

[0155] As shown in FIG. 15, the processes from step S10 to step S13A are the same as the processes from step S10 to step S13 in the second control example described with reference to FIG.

[0156] Next, processor 113 updates the "size information" relating to the size of the subject and the "speed information" relating to the moving speed of the subject that were set up until then based on the detection result of the subject detected in the detection process of step S13A and the distance from camera 110 to the subject (step S13B). The assumed "size information" and "speed information" set at the start of tracking of the subject are updated to "size information" and "speed information" calculated based on the detection results of the subject that are sequentially detected and the distance to the subject after tracking starts. This makes it possible to set a more optimized analysis range for the subject.

[0157] The processing of steps S14 and S15 in Fig. 15 is similar to the processing of steps S14 and S15 in the second control example described in Fig. 5, and therefore description thereof will be omitted. Note that in this modification, the size and speed of the subject are updated based on the detection result and the subject distance, but this is not limiting. For example, the size and speed of the subject may be updated by the user inputting the size and speed of the subject.

[0158] <Modification (Part 2) of Setting the Analysis Range> FIG. 16 is a flowchart showing a modification (Part 2) of the third setting of the analysis range (FIG. 8).

[0159] As shown in Figure 16, in this modified example, the processor 113 performs a process of acquiring distance information regarding the distance to a specific object from the radar 130 (step S41A), and then determines whether or not the distance information was acquired (step S41B).

[0160] If the distance information can be acquired in step S41B (step S41B: Yes), the processor 113 proceeds directly to step S42. On the other hand, if the distance information cannot be acquired from the radar 130 in step S41B (step S41B: No), the processor 113 acquires distance information based on the analysis result (step S41C) and then proceeds to step S42. If the distance information cannot be acquired from the radar 130, this means that the radar 130 cannot detect (e.g., loses sight of) a specific object (e.g., the drone 50).

[0161] The distance information based on the analysis results is, for example, distance information to the subject calculated based on the analysis results (e.g., the size of the image area in which the subject is detected) obtained by analyzing the analysis range in the subject detection process, the focal length (angle of view) of the camera 110, and "size information" regarding the size of the subject.

[0162] The processes from step S42 to step S47 in Fig. 16 are the same as the processes from step S42 to step S47 in the third setting example described in Fig. 8. According to this modification, even if the radar 130 cannot detect a specific object, for example, it is possible to use distance information based on the analysis results, and therefore it is possible to appropriately set the analysis range of the subject.

[0163] 17 is a diagram showing an example of the correspondence relationship between the type of subject and its expected size and expected speed. As shown in Fig. 17, the camera 110 may store in advance, for example, in the memory 35 of the computer 19, correspondence information 81 between the expected type of subject and the expected size and speed of the subject.

[0164] For example, when detecting and tracking a specific subject, the user selects the type of subject (e.g., drone (model A)) from the correspondence information 81. The processor 113 acquires the estimated size and estimated speed associated with the selected type of subject from the correspondence information 81, and sets them as the size and speed of the subject to be used in setting the analysis range of the captured image.

[0165] This allows the size and speed of the subject to be set efficiently. Note that the setting of the type of subject is not limited to a user selecting operation. For example, the processor 113 may identify the type of subject by analyzing the image captured by the camera 110, and set the identified type of subject.

[0166] <First Example of Shifting Analysis Range> FIG. 18 is a flowchart showing a first example of shifting the position of the determined analysis range.

[0167] As shown in FIG. 18, the processes from step S41 to step S46 are the same as the processes from step S41 to step S46 in the third setting example described with reference to FIG.

[0168] After the analysis range is determined in step S45 or step S46, the processor 113 determines the shift position of the analysis range based on the results of past analysis (step S47A). "Based on the results of past analysis" refers to, for example, the position within the analysis range at which the subject was detected in the previous subject detection process within the analysis range.

[0169] Determining the shift position of the analysis range means shifting the analysis range so that the detected subject is located, for example, approximately in the center of the analysis range, or shifting the analysis range so that the subject is located slightly further in the direction of movement than the center of the analysis range. This makes it difficult for the subject to deviate from the analysis range, making it possible to track subjects that are moving quickly.

[0170] Next, the processor 113 sets the determined analysis range in, for example, the memory 35 of the computer 19 (step S47B).

[0171] <Second Example of Shifting Analysis Range> FIG. 19 is a flowchart showing a second example of shifting the position of the determined analysis range.

[0172] As shown in FIG. 19, the processes from step S41 to step S46 are the same as the processes from step S41 to step S46 in the third setting example described with reference to FIG.

[0173] After the analysis range is determined in step S45 or step S46, the processor 113 determines the shift position of the analysis range based on the direction information from the radar 130 (step S47C). The direction information from the radar 130 is "directional information" relating to the direction of a specific object detected by the radar 130.

[0174] Determining the shift position of the analysis range means, for example, shifting the analysis range so that the subject is positioned approximately in the center of the analysis range, as in the first shift example in Fig. 18. This makes it possible to track a subject that moves at a fast speed, as in the first shift example.

[0175] Next, the processor 113 sets the determined analysis range in, for example, the memory 35 of the computer 19 (step S47D).

[0176] <Example of Shifting Analysis Range 61 with Respect to Imaging Range 60> Figure 20 is a diagram showing an example of shifting the analysis range 61 with respect to the imaging range 60 of the camera 110. As shown in Figure 20, as a result of the subject detection process, it is assumed that the subject to be tracked (drone 50) is detected at a position to the right in the dashed analysis range 61 that is set in the center of the imaging range 60. In this case, the processor 113 shifts the analysis range 61 to the right so that the detected drone 50 is located approximately in the center of the analysis range 61, and determines the shift position of the analysis range to the position of the solid analysis range 61.

[0177] <Storage medium for control program> In each of the above imaging controls, an example has been described in which the control program of each embodiment is stored in the storage 36 of the computer 19 and the CPU 37 of the computer 19 executes the control program in the memory 35, but the technology of the present disclosure is not limited to this.

[0178] Fig. 21 is a diagram showing an example of how a control program for image capture control is installed from a storage medium on which the control program is stored into the computer 19 of the camera (image capture device) 110. As an example, as shown in Fig. 21, a control program 221 may be stored in a storage medium 220, which is a non-transitory storage medium. In the example shown in Fig. 21, the control program 221 stored in the storage medium 220 is installed into the computer 19, and the CPU 37 executes the above-described processes in accordance with the control program 221.

[0179] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0180] This application is based on a Japanese patent application filed on August 28, 2023 (Patent Application No. 2023-138033) and a Japanese patent application filed on October 16, 2023 (Patent Application No. 2023-178219), the contents of which are incorporated by reference into this application.

[0181] 15 Optical system 15A Objective lens 15B Lens group 15B1 Anti-vibration lens 15B2 Zoom lens 17, 21 Lens actuator 19 Computer 22 BIS driver 23 OIS driver 25 Imaging element 25A Light receiving surface 27 Imaging element actuator 28 Lens driver 29, 45 Correction mechanism 31 DSP 32 Image memory 33 Electronic shake correction unit 34 Communication I / F 35 Memory 36 Storage 37 CPU 38 Bus 39, 47 Position sensor 40 Shake amount detection sensor 43 UI device 43A Reception device 43B Display 50 Drone 60 Imaging range 61 Analysis range 71 Display screen 72 Message 73 Red frame 81 Corresponding information 100 Imaging system 110 Camera 111 Lens 112 Sensor 113 Processor 113a Image cutting / resizing unit 113b Object detection unit 113c Rotation control unit 113d Distance acquisition unit 113e Image analysis range determination unit 120 Rotation device 130 Radar 220 Storage medium 221 Control program

Claims

1. A control device included in an imaging system comprising an imaging device, a rotation device that rotates the imaging device and is controlled based on the results of analysis of an image captured by the imaging device, and a distance measuring device, the control device comprising a processor, wherein the processor sets the range of analysis in the captured image based on information regarding the distance to a specific object obtained from the distance measuring device and at least one of information regarding the size or information regarding the speed of the specific object.

2. A control device according to claim 1, wherein the specific object is a specific subject included in the captured image.

3. A control device according to claim 1, wherein said processor performs said analysis.

4. A control device according to claim 1, wherein the processor controls the turning device based on the result of the analysis.

5. A control device according to claim 1, wherein the distance measuring device is a device for obtaining information relating to a direction.

6. A control device according to claim 1, wherein the distance measuring device is a radio wave detector.

7. A control device as described in claim 5, wherein the processor controls the rotation device based on the detection result of the specific object by the analysis, and when the detection result satisfies a first condition, controls the rotation device based on information regarding the direction.

8. A control device according to claim 7, wherein the first condition is a condition relating to whether or not the specific object is detected.

9. A control device according to claim 8, wherein the first condition is a condition in which the specific object is not detected.

10. A control device as described in claim 1, wherein the processor sets the analysis range based on a first range based on information in the captured image regarding the distance to the specific object and information regarding the size of the specific object, and a second range based on information in the captured image regarding the distance to the specific object and information regarding the speed of the specific object.

11. A control device according to claim 10, wherein the processor sets the analysis range to a range narrower than the first range and wider than the second range.

12. A control device according to claim 11, wherein said processor sets said first range as the range of said analysis if said first range is narrower than said second range.

13. The control device of claim 1, wherein the focal length of the imaging device is variable, and the processor controls the focal length based on the extent of the analysis.

14. A control device according to claim 13, wherein the processor controls the focal length so that the imaging range of the imaging device is wider than the range of the analysis.

15. A control device according to claim 1, wherein the processor causes a display device to display an image obtained by cropping the captured image based on a result of the analysis.

16. A control device according to claim 15, wherein the processor cancels the cropping and causes the captured image to be displayed on the display device when the specific object is not detected by the analysis.

17. A control device according to claim 1, wherein the processor notifies a user when the specific object is not detected by the analysis.

18. The control device of claim 1, wherein the processor selects the mode of analysis from among a plurality of analysis modes based on information regarding the distance of the particular object.

19. A control device according to claim 1, wherein the processor sets a range of the analysis based on an environment in which the imaging device captures images.

20. A control device according to claim 1, wherein the processor updates at least one of information relating to the size or information relating to the speed of the specific object used to set the range of the analysis, based on information relating to the distance of the specific object and the results of the analysis.

21. A control device as described in claim 1, wherein, when information regarding the distance to the specific object has not yet been obtained from the distance measuring device, the processor sets the range of the analysis based on information regarding the distance to the specific object based on the results of the analysis and at least one of information regarding the size or information regarding the speed of the specific object.

22. A control device according to claim 1, wherein the processor updates at least one of information relating to the size or information relating to the speed of the specific object used to set the range of the analysis based on information acquired from the distance measuring device.

23. A control device as described in claim 1, wherein the processor acquires correspondence information between a type of a specific object and at least one of information relating to the size or speed of the specific object, and acquires at least one of information relating to the size or speed of the specific object to be used for setting the range of the analysis based on the type of the specific object and the correspondence information.

24. The control device according to claim 1, wherein the processor sets a position of the range of the analysis based on the result of the analysis.

25. A control device according to any one of claims 1 to 22, wherein the processor sets the position of the analysis range based on information acquired from the distance measuring device.

26. A control method for an imaging system comprising an imaging device, a rotation device that is controlled based on the results of analysis of an image captured by the imaging device and rotates the imaging device, and a distance measuring device, wherein a processor of a control device included in the imaging system sets the range of analysis in the captured image based on information regarding the distance to a specific object obtained from the distance measuring device and at least one of information regarding the size or information regarding the speed of the specific object.

27. A control program for an imaging system comprising an imaging device, a rotation device that rotates the imaging device under control based on the results of analysis of an image captured by the imaging device, and a distance measuring device, the control program causing a processor of a control device included in the imaging system to execute processing to set the range of analysis in the captured image based on information regarding the distance to a specific object obtained from the distance measuring device and at least one of information regarding the size or speed of the specific object.

Images